Skip to content

fix: handle tool call parse errors gracefully in server#899

Open
shanemmattner wants to merge 1 commit intoml-explore:mainfrom
shanemmattner:fix/handle-tool-parse-errors
Open

fix: handle tool call parse errors gracefully in server#899
shanemmattner wants to merge 1 commit intoml-explore:mainfrom
shanemmattner:fix/handle-tool-parse-errors

Conversation

@shanemmattner
Copy link

Problem

When a model produces tool call arguments with invalid JSON (e.g. raw regex like \d+, unescaped backslash paths), json.loads() in the tool parser raises JSONDecodeError. This exception is unhandled in parse_tools(), crashes the HTTP handler, and drops the client connection mid-response.

Affected parsers: json_tools.py, mistral.py, and any parser that calls json.loads() or raises ValueError on malformed input.

Fix

Wrap the ctx.tool_parser() call in parse_tools() with a try/except that catches JSONDecodeError, ValueError, and KeyError. On failure, the server logs a warning and skips the malformed tool call instead of crashing.

This is a 4-line change at the call site in server.py, so it covers all current and future tool parsers.

Reproduction

  1. Run mlx_lm.server with a model that supports tool use (e.g. Qwen3-32B)
  2. Send a prompt that causes the model to generate regex patterns in tool arguments
  3. Server crashes with JSONDecodeError: Invalid \escape

Testing

Tested with Qwen3-32B-4bit across 4 prompt categories (regex-heavy, backslash paths, newline escapes, clean JSON) — all pass, server stays up, malformed tool calls are logged and skipped.

Wrap tool parser calls in try/except so that malformed tool call
JSON from the model logs a warning instead of crashing the server.

Models (e.g. Qwen3) sometimes produce invalid JSON escape sequences
in tool arguments (raw \d+, \n, backslash-heavy paths). The bare
json.loads() in tool parsers raises JSONDecodeError, which propagates
unhandled through parse_tools() and crashes the HTTP handler, dropping
the client connection mid-response.

This fix catches parse errors at the call site in server.py rather
than in each individual parser, so it covers all current and future
tool parsers (json_tools, mistral, qwen3_coder, etc.).
@shanemmattner shanemmattner force-pushed the fix/handle-tool-parse-errors branch from 6c61e7b to e02c5ef Compare February 16, 2026 05:30
@shanemmattner shanemmattner marked this pull request as ready for review February 16, 2026 05:39
@shanemmattner
Copy link
Author

Update: I was testing this on mlx-lm 0.25.0. After upgrading to 0.30.7, I see that some parsers (like glm47.py) now have fallback logic via ast.literal_eval().

However, json_tools.py and mistral.py still have bare json.loads() with no error handling. This PR provides a safety net at the call site that covers all parsers (current and future).

Tested on 0.30.7 — the server-level try/except still catches exceptions from parsers without fallbacks. Happy to close this if you'd prefer a different approach.

@awni
Copy link
Member

awni commented Feb 16, 2026

Is there a way to send a response that the tool call did not parse properly rather than just logging it and silently continuing? Does the OpenAI API spec do anything in those cases?

Like it might be better to return a "tool call failed to parse" error so that it can be given as input to the model on the next request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants