Structural generation and function calling / tool use #771
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ocess_inference_result`
…o more closely align with OpenAI
…tion from the ChatGPT API
…und for textual stop sequence matches
This removes direct references the internals of BufferedOutput to allow for better abstraction
…, read from the ChatCompletionRequest, along with tests
joshuacoles
force-pushed
the
structural-generation-and-function-calling
branch
from
51db6af to
f95eb91
Compare
ToolBehaviour is a custom request property which allows configuration of if tool generation should be guided (ie. with a grammar) or left entirely to the model and if it should be parsed (ie. returned as raw tokens for client consumption or parsed into JSON in the response).
…etionRequest and start to implement tool completions
joshuacoles
force-pushed
the
structural-generation-and-function-calling
branch
2 times, most recently
from
45a647e to
9371027
Compare
…n other places. More investigation needed. See: https://github.com/ollama/ollama/blob/4aeb67ef4c0c734e0077af39e0b8be1252662cc0/template/index.json#L95 for more
joshuacoles
force-pushed
the
structural-generation-and-function-calling
branch
from
9371027 to
664e70f
Compare
|
I am going to continue to cleanup the git history to make it easier to review commit wise but I would say this is now feature complete. There are more things to add but you've got to stop somewhere and this is enough for our immediate use case. |
…ced in git cleanup
palios-taey
added a commit
to palios-taey/exo
that referenced
this pull request
Oct 20, 2025
Implements complete server-side parsing and formatting of tool calls to match OpenAI API specification. Fixes exo-explore#293. Changes: - Add parse_tool_calls() function to parse <tool_call> XML tags from model output - Modify generate_completion() to detect and format tool calls in responses - Return proper tool_calls array with OpenAI-compliant structure - Set finish_reason to "tool_calls" when tools are invoked - Support both streaming and non-streaming responses - Handle parallel tool calling (multiple tools in one response) - Generate unique call IDs server-side (call_<random>) - Ensure arguments field is always a JSON string (not object) Implementation details: - Reuses existing XML tag pattern from examples/function_calling.py - Minimal changes to chatgpt_api.py (focused on response generation) - Backwards compatible (no changes when tools not provided) - Works with all existing tokenizer chat templates that support tools Testing: - Added comprehensive unit tests in test_parse_simple.py - All 5 tests pass (single/parallel/no tools/dict conversion/OpenAI format) - Added new example: examples/function_calling_openai_compliant.py This implementation is cleaner and more focused than the stalled PR exo-explore#771 (59 commits, 35 files). We achieve the same functionality with minimal changes to core API logic.
8 tasks
AlexCheema
force-pushed
the
main
branch
2 times, most recently
from
a39f85b to
56f783b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a combined PR for structural generation and OpenAI spec compatible function calling. A description of its contents can be found in: #771 (comment).
This builds off #734 and contains those commits