[Bug]: litellm Connection Error with ADK

[kuri-leo]

What happened?

Hi folks,

I'm running into an issue using litellm (v1.74.7 and v1.74.8) with google-adk (v1.8.0), when connecting to a self-hosted SGLang instance.

I'm initializing LiteLLM like this:

LiteLlm(
    model="hosted_vllm/Qwen2.5-7B-Instruct",  # also tried "openai/Qwen2.5-7B-Instruct"
    api_base="http://192.168.125.4:18008/v1",
    api_key="sk-123456",
)

Everything runs smoothly at first, but after ~20 minutes of parallel ADK calls, I get the following error:

litellm.InternalServerError: InternalServerError: Hosted_vllmException - Connection error

I’ve ruled out network issues, server instability, and client-side bugs, everything works fine when using the official openai client instead of litellm.

I also tested with different concurrency levels and timeouts, but the error still shows up after some runtime. Unfortunately, enabling debug mode didn’t yield any useful logs.

Hope this helps track down the issue. Let me know if I can provide more info

Potential related issues:

• [Bug]: LM Studio not working with LiteLLM Proxy #11733

Relevant log output

litellm.InternalServerError: InternalServerError: Hosted_vllmException - Connection error

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.74.7, v1.74.8

Twitter / LinkedIn details

No response

[kuri-leo]

Related to: #12443

While debugging, I noticed that completed requests are not being properly closed. This results in a large number of connections stuck in the FIN_WAIT2 state, which eventually exhausts available ports.

The LLM server is SGLang:v0.4.9.post2-cu126:

docker run --gpus '"device=0,1,2,3,4,5,6,7"' \
    -p "${PORT_NUM}:${PORT_NUM}" \
    --ipc=host \
    --rm -itd \
    --name "${CONTAINER_NAME}" \
    lmsysorg/sglang:v0.4.9.post2-cu126 \
    python3 -m sglang.launch_server \
    --host 0.0.0.0 \
    --port "${PORT_NUM}" \
    --model-path ${MODEL_PATH} \
    --served-model-name ${MODEL_NAME} \
    --trust-remote-code \
    --dtype bfloat16 \
    --dp 8 \
    --tp 1 \
    --mem-fraction-static 0.6