Claude 3.7 hosted on databricks, does not work in Agent or Edit mode

[harkamals]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: macOS
- Continue version: 1.0.11
- IDE version: 1.100.3
- Model: databricks-claude-3-7-sonnet
- config:
  
name: Local Assistant
version: 1.0.0
schema: v1
models:
  - name: Claude 3.7
    provider: openai
    model: databricks-claude-3-7-sonnet
    apiBase: https://redacted.azuredatabricks.net/serving-endpoints
    apiKey: redacted
    capabilities:
      - tool_use
      - image_input
    requestOptions:
      verifySsl: false
      proxy: http://redacted:443
  - name: GPT 4o
    provider: openai
    model: gpt4o
    apiBase: https://redacted.azuredatabricks.net/serving-endpoints
    apiKey: redacted
    capabilities:
      - tool_use
      - image_input
    requestOptions:
      verifySsl: false
      proxy: http://redacted:443
context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  - provider: codebase

Description

Our models are hosted on azuredatabricks.net

OpenAI model works in Chat and Agent mode

• Edit mode does not work
  "{\"external_model_provider\":\"openai\",\"external_model_or_query_router_error\":\"Unsupported data type\\n\"}"

Claude model works in Chat mode only.

• Agent mode does not work
  400 status code (no body)
• Edit mode does not work
  "{\"external_model_provider\":\"openai\",\"external_model_or_query_router_error\":\"Unsupported data type\\n\"}"

To reproduce

No response

Log output

[harkamals]

Appreciate some guidance or way forward. Thank you in anticipation.

[sestinj]

It seems like there is something about the azure databricks solution that is not fully OpenAI-compatible. Are you able to take a look in the logs (https://docs.continue.dev/troubleshooting#check-the-logs) as a next step, or see if there is a curl request with tool use that works successfully for you?

[harkamals]

--> Yes, curl works ok, with AzureDatabricks.net hosted claude sonnet and gpt4o models.

Could you also share full list of environment variables, especially to suppress invalid ssl certs, I want to test if this could be an issue, as corporate proxy rewrites ssl. Thanks.

`

export DATABRICKS_TOKEN=redacted

curl https://redacted.azuredatabricks.net/serving-endpoints/databricks-claude-3-7-sonnet/invocations
-X POST
-u token:$DATABRICKS_TOKEN
-H "Content-Type: application/json"
-d '{"messages":[{"role":"user","content":"What is an LLM agent?"}]}'
{"model":"eu.anthropic.claude-3-7-sonnet-20250219-v1:0","choices":[{"message":{"role":"assistant","content":"# LLM Agent\n\nAn LLM (Large Language Model) agent is an AI system built around a large language model that can:\n\n1. Perform tasks autonomously or semi-autonomously\n2. Use reasoning capabilities to solve problems\n3. Interact with external tools and APIs (like web search, coding environments, or databases)\n4. Plan multi-step processes to achieve goals\n5. Learn from feedback and adapt its approach\n\nLLM agents extend beyond simple question-answering by combining language models with decision-making frameworks, allowing them to take actions based on their understanding of requests and available resources. They're designed to be more interactive and capable of sustained, goal-oriented behavior than basic language models alone."},"index":0,"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":155,"total_tokens":170},"object":"chat.completion","id":"msg_bdrk_01Gtr4HuZpmiof3NioZxirrW","created":1749524172}>

`

`

curl https://redacted.azuredatabricks.net/serving-endpoints/gpt4o/invocations
-X POST
-u token:$DATABRICKS_TOKEN
-H "Content-Type: application/json"
-d '{"messages":[{"role":"user","content":"What is an LLM agent?"}]}'

{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"An LLM agent, or a Large Language Model agent, refers to a system or application that leverages a large language model to perform specific tasks or functions. These models, like OpenAI's GPT-3 or GPT-4, are designed to understand and generate human-like text based on vast amounts of training data. Here’s a more detailed explanation:\n\n1. Components:\n - Large Language Model (LLM): The core of an LLM agent, these are neural networks trained on diverse text data, capable of performing tasks ranging from text generation to comprehension, translation, summarization, and more.\n \n2. Tasks:\n - LLM agents can be tailored for various applications including conversational agents (chatbots), content creation tools, translation services, coding assistants, and more.\n\n3. Integration:\n - In many applications, LLMs are integrated with other software systems or modules. This could involve connecting the LLM with databases, APIs, or user interfaces to create a seamless and functional end-user application.\n\n4. Autonomy:\n - Some LLM agents are designed to operate with a degree of autonomy, making decisions based on predefined rules or learning from prior interactions to improve over time.\n\n5. Use Cases:\n - Examples include virtual assistants capable of managing schedules and emails, chatbots for customer service, or tools that can automatically generate reports or articles based on minimal input.\n\nOverall, LLM agents combine the language-processing power of large language models with specific functionalities and interfaces tailored for particular applications, offering sophisticated and versatile capabilities across various domains.","refusal":null,"role":"assistant"}}],"created":1749524179,"id":"chatcmpl-BgjRr9AsOCQLYNQotxFEFd8PdKc3z","model":"gpt-4o-2024-08-06","object":"chat.completion","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"system_fingerprint":"fp_ee1d74bde0","usage":{"completion_tokens":325,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens":14,"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"total_tokens":339}}

`

[barrel-roll-42]

Any updates on this issue?
I'm having the same issues with agent mode with utilizing databricks hosted Claude 4 sonnet and Llama4 maverick.

GPT 4o via databricks works fine for agent mode (I haven't tried edit)