Skip to content

feat: add context_length to /v1/models response#933

Open
lichengzhe wants to merge 1 commit intoml-explore:mainfrom
lichengzhe:feat/models-context-length
Open

feat: add context_length to /v1/models response#933
lichengzhe wants to merge 1 commit intoml-explore:mainfrom
lichengzhe:feat/models-context-length

Conversation

@lichengzhe
Copy link

Summary

Include context_length in the /v1/models endpoint response by reading max_position_embeddings from each model's config.json.

Motivation

Many OpenAI-compatible API gateways and proxy tools (e.g. one-api, new-api) read context_length from the /v1/models response to automatically determine a model's context window size. Without this field, users must manually configure context limits for each model, which is error-prone and tedious.

Related: #966 (KeyError on context_length)

Changes

  • Added a get_context_length() helper that reads max_position_embeddings from a HuggingFace cached model's config.json
  • Updated the HF cache model listing to include context_length when available
  • Updated the local model path listing to also read and include context_length
  • Handles models that nest the value under text_config (e.g. some vision-language models)
  • Gracefully falls back (omits the field) if config.json is missing or unreadable

Example response

{
  "object": "list",
  "data": [
    {
      "id": "RepublicOfKorokke/Qwen3.5-35B-A3B-mlx-lm-mxfp4",
      "object": "model",
      "created": 1740000000,
      "context_length": 262144
    }
  ]
}

Testing

Tested locally with Qwen3.5-35B-A3B-mlx-lm-mxfp4 — confirmed context_length: 262144 appears in the response.

Read max_position_embeddings from each model's config.json and include
it as context_length in the /v1/models endpoint response. This helps
API gateways (e.g. one-api, new-api) and clients automatically discover
the model's context window size without manual configuration.

Handles both HuggingFace cached models and local path models, including
models that nest the value under text_config.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant