Unable to connect to hugging face inference providers

[davidpangsw]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:macOS
- Continue version: v1.1.37
- IDE version:VSCode 1.92
- Model: `deepseek/deepseek-v3-0324`, or others
- config:
  
name: Local Assistant
version: 1.0.0
schema: v1
models:
  - name: chat-model
    provider: huggingface-inference-api
    model: deepseek/deepseek-v3-0324
    apiBase: https://router.huggingface.co/novita/v3/openai
    apiKey: hf_XXXXX
    roles:
      - chat
      - edit
      - autocomplete
context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  
  OR link to assistant in Continue hub:

Description

It cannot integrate with the inference providers described in hugging face docs about inference providers.
I have tried out different routes and models, but it still gives:

Error streaming edit diffs: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

I confirmed my api key is correct.

To reproduce

1. get an proper API key from hugging face
2. go to config.yaml and set the config as above
3. Send any prompt

Log output

[Patrick-Erichsen]

Hi @davidpangsw , have you tried using the model in Chat? It looks like the error message you sent is specifically from Edit.

It should be working just fine with Edit however so that's still a bug, just curious if it's working for other modes.

Additionally, if you can share the logs (specifically prompt logs) that would be helpful.

[davidpangsw]

Hi @davidpangsw , have you tried using the model in Chat? It looks like the error message you sent is specifically from Edit.

It should be working just fine with Edit however so that's still a bug, just curious if it's working for other modes.

Additionally, if you can share the logs (specifically prompt logs) that would be helpful.

Thank you for your time. Here is the log:

console.ts:137 [Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "72b0eb0b-0ac0-4473-8434-bb18c2994854",
    "messageType": "llm/streamChat",
    "data": {
      "completionOptions": {},
      "title": "chat-model",
      "messages": [
        {
          "role": "system",
          "content": "<important_rules>\n  You are in chat mode.\n\n  If the user asks to make changes to files offer that they can use the Apply Button on the code block, or switch to Agent Mode to make the suggested updates automatically.\n  If needed consisely explain to the user they can switch to agent mode using the Mode Selector dropdown and provide no other details.\n\n  Always include the language and file name in the info string when you write code blocks.\n  If you are editing \"src/main.py\" for example, your code block should start with '```python src/main.py'\n\n  When addressing code modification requests, present a concise code snippet that\n  emphasizes only the necessary changes and uses abbreviated placeholders for\n  unmodified sections. For example:\n\n  ```language /path/to/file\n  // ... existing code ...\n\n  {{ modified code here }}\n\n  // ... existing code ...\n\n  {{ another modification }}\n\n  // ... rest of code ...\n  ```\n\n  In existing files, you should always restate the function or class that the snippet belongs to:\n\n  ```language /path/to/file\n  // ... existing code ...\n\n  function exampleFunction() {\n    // ... existing code ...\n\n    {{ modified code here }}\n\n    // ... rest of function ...\n  }\n\n  // ... rest of code ...\n  ```\n\n  Since users have access to their complete file, they prefer reading only the\n  relevant modifications. It's perfectly acceptable to omit unmodified portions\n  at the beginning, middle, or end of files using these \"lazy\" comments. Only\n  provide the complete file when explicitly requested. Include a concise explanation\n  of changes unless the user specifically asks for code only.\n\n</important_rules>"
        },
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "Hello, who are you?"
            }
          ]
        },
        {
          "role": "assistant",
          "content": ""
        }
      ]
    }
  }
}

SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

And screen-cap:

I tried both chat and edit. I guess (but not sure) the problem is about the compatibility to the endpoints from inference providers

[Patrick-Erichsen]

(accidentally closed/reopened)

Hm, yeah it seems like this is likely a formatting/streaming error on our end. Not sure how soon we'll be able to prioritize this, so I added a good first issue label if available to help out in the meantime.