fix: handle tool call parse errors gracefully in server#899
fix: handle tool call parse errors gracefully in server#899shanemmattner wants to merge 1 commit intoml-explore:mainfrom
Conversation
Wrap tool parser calls in try/except so that malformed tool call JSON from the model logs a warning instead of crashing the server. Models (e.g. Qwen3) sometimes produce invalid JSON escape sequences in tool arguments (raw \d+, \n, backslash-heavy paths). The bare json.loads() in tool parsers raises JSONDecodeError, which propagates unhandled through parse_tools() and crashes the HTTP handler, dropping the client connection mid-response. This fix catches parse errors at the call site in server.py rather than in each individual parser, so it covers all current and future tool parsers (json_tools, mistral, qwen3_coder, etc.).
6c61e7b to
e02c5ef
Compare
|
Update: I was testing this on mlx-lm 0.25.0. After upgrading to 0.30.7, I see that some parsers (like However, Tested on 0.30.7 — the server-level try/except still catches exceptions from parsers without fallbacks. Happy to close this if you'd prefer a different approach. |
|
Is there a way to send a response that the tool call did not parse properly rather than just logging it and silently continuing? Does the OpenAI API spec do anything in those cases? Like it might be better to return a "tool call failed to parse" error so that it can be given as input to the model on the next request. |
Problem
When a model produces tool call arguments with invalid JSON (e.g. raw regex like
\d+, unescaped backslash paths),json.loads()in the tool parser raisesJSONDecodeError. This exception is unhandled inparse_tools(), crashes the HTTP handler, and drops the client connection mid-response.Affected parsers:
json_tools.py,mistral.py, and any parser that callsjson.loads()or raisesValueErroron malformed input.Fix
Wrap the
ctx.tool_parser()call inparse_tools()with a try/except that catchesJSONDecodeError,ValueError, andKeyError. On failure, the server logs a warning and skips the malformed tool call instead of crashing.This is a 4-line change at the call site in
server.py, so it covers all current and future tool parsers.Reproduction
mlx_lm.serverwith a model that supports tool use (e.g. Qwen3-32B)JSONDecodeError: Invalid \escapeTesting
Tested with Qwen3-32B-4bit across 4 prompt categories (regex-heavy, backslash paths, newline escapes, clean JSON) — all pass, server stays up, malformed tool calls are logged and skipped.