Does TensorrtLLM support v1/response api?

[sowmen]

Hi, does trtllm support the openai v1/response api, if so what arguments are supported - such as reasoning effort, structured output, etc.

[Advait251206]

The TensorRT-LLM server stack does not natively implement the OpenAI v1/responses API as a first-class, fully compatible endpoint in the same way the OpenAI platform does.

Instead, :contentReference[oaicite:0]{index=0} (typically via :contentReference[oaicite:1]{index=1} or its reference server) exposes OpenAI-compatible REST APIs in a partial / evolving form, primarily aligned with:

• /v1/chat/completions
• /v1/completions (legacy-style)
• streaming generation endpoints (SSE-style)

---

1. Support for /v1/responses

At the time of writing:

• ❌ No full native support for /v1/responses
• ❌ No official parity with OpenAI “Responses API” semantics
• ⚠️ Some forks or custom gateways may emulate it, but it is not part of core TensorRT-LLM serving

The main reason is that /v1/responses is a newer OpenAI abstraction layer that includes:

• multi-step tool orchestration
• structured reasoning pipelines
• richer response objects

These features are outside the scope of the current TensorRT-LLM inference server design.

---

2. What is actually supported

The typical supported OpenAI-compatible surface in TensorRT-LLM includes:

Supported (common)

• messages-based chat format
• temperature, top_p, max_tokens
• stream (token streaming)
• stop sequences
• basic logit control (depending on backend)

Partially supported / backend-dependent

• JSON/structured output (via prompt engineering or constrained decoding in limited setups)
• logits biasing / logit processors (varies by deployment)
• tool calling (not native; usually emulated externally)

---

3. Unsupported or not standardized

❌ “reasoning_effort”

• Not part of TensorRT-LLM API schema
• No native mapping to inference engine behavior
• Would require external orchestration logic

❌ Structured output (strict schema enforcement)

• No OpenAI-style JSON schema enforcement layer in core server
• Can be approximated using:
  • constrained decoding (experimental)
  • regex/grammar-based decoding (limited support depending on build)

❌ /v1/responses features

• multi-turn tool orchestration
• built-in reasoning traces
• response object composition (annotations, tool steps, etc.)

These are outside the scope of the current inference-only engine.

---

4. Why this limitation exists

The design of :contentReference[oaicite:2]{index=2} focuses on:

• High-performance token generation
• CUDA/TensorRT-optimized execution graphs
• Low-latency inference serving

Whereas /v1/responses is a higher-level orchestration API, which includes:

• agentic workflows
• tool execution graphs
• structured multi-step reasoning pipelines

Those responsibilities typically belong to an application layer, not an inference engine.

---

5. Practical workaround

If you want /v1/responses-like behavior on TensorRT-LLM:

Option A — External orchestration layer (recommended)

Build a gateway that:

• translates /v1/responses → /v1/chat/completions
• handles tool calling externally
• manages structured output validation

Option B — Prompt-based structured output

• enforce JSON schema via prompt engineering
• optionally validate output downstream

Option C — Use OpenAI-compatible frontends

Some community servers add:

• partial /v1/responses emulation
• but these are not official TensorRT-LLM features

---

6. Conclusion

• ❌ TensorRT-LLM does not natively support OpenAI /v1/responses
• ✔ It supports a subset of OpenAI-style chat/completions APIs
• ⚠ Advanced fields like reasoning control and structured response orchestration must be handled outside the inference engine
• 🧠 /v1/responses is fundamentally an orchestration-layer API, not an inference-layer API, which is why it is not directly implemented

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206