VSCode: Autocomplete with AWS Bedrock triggers an error toast

[maximelebastard]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: macOS
- Continue version: 1.1.32
- IDE version: 1.100.2
- Model: any
- config: 
  
%YAML 1.1
---
version: 1.0.16
schema: v1

bedrock: &bedrock
  provider: bedrock
  env:
    region: eu-central-1
    profile: bedrock

models:
  - name: (aws) Anthropic Claude 3.5 Sonnet
    <<: *bedrock
    model: anthropic.claude-3-5-sonnet-20240620-v1:0
    roles:
      - chat
      - edit
      - apply
  - name: (aws) Anthropic Claude 3.7 Sonnet
    <<: *bedrock
    model: eu.anthropic.claude-3-7-sonnet-20250219-v1:0
    roles:
      - chat
      - edit
  - name: (aws) Anthropic Claude 3 Haiku
    <<: *bedrock
    model: anthropic.claude-3-haiku-20240307-v1:0
    roles:
      - chat
      - autocomplete

  - name: (aws) Cohere Rerank
    <<: *bedrock
    model: cohere.rerank-v3-5:0
    roles:
      - rerank

  - name: (aws) Cohere Embed Multilingual
    <<: *bedrock
    model: cohere.embed-multilingual-v3
    roles:
      - embed
      
  - name: (local) Qwen 2.5 autocomplete
    provider: ollama
    model: qwen2.5-coder:3b
    roles:
      - chat

context:
  - provider: file
  - provider: code
  - provider: open
  - provider: folder
  - provider: clipboard
  - provider: diff
  - provider: terminal
  - provider: docs
  - provider: web
    params:
      n: 5
  - provider: codebase
  - provider: tree
  - provider: url

prompts:
  - name: commit
    description: Create a commit message from staged files
    prompt: "{{input}}"

docs:
  - name: Symfony
    startUrl: https://symfony.com/doc/6.4/index.html

defaultCompletionOptions:
  maxTokens: 4096

Description

When using an AWS Bedrock model as the autocomplete model, typing in a file triggers frequent error toasts about the previous request having been canceled.
This is probably because typing new text cancels the ongoing autocomplete request to create a new one.

To reproduce

1. Set Haiku 3 model from Bedrock as the autocomplete model
2. Start to type something in a file
3. Wait for a second
4. Start typing again
5. The toast "Failed to communicate with Bedrock API: Request aborted" appears
6. The toast "AWS Bedrock stream error (ECONNRESET): aborted" from Continue extension appears on the right hand side of VSCode

Log output

[chezsmithy]

Autocomplete requires a FIM enabled model to really work. Currently Bedrock doesn't have any FIM models in its catalog that really work well. We've imported CodeLlama for this purpose but it's not the greatest. I'm holding out for mistral to add codestral to bedrock.

[jmoughon]

Reviewing your config:

change Qwen 2.5 autocomplete to have:
roles:
- autocomplete

remove autocomplete role from Anthropic Claude 3 Haiku

[maximelebastard]

@chezsmithy FIM models may be more accurate, but Haiku mainly works - the error prompt is useless and only related to canceled http requests - probably due to bad error catching. My ticket is not here to report any completion issue, but an http / aws sdk usage issue.

@jmoughon Thanks i'll give it a try, but as there is a model selector (the small box icon over the prompt area) I thought the model used would be the one selected there. And it is Haiku, not Ollama Qwen.

[jmoughon]

@maximelebastard Understood. You need to update your config to allow Qwen:

- name: (local) Qwen 2.5 autocomplete
    provider: ollama
    model: qwen2.5-coder:3b
    roles:
      - autocomplete