fix(param): emit extra fields in sorted key order - #901
Open
gtoxlili wants to merge 1 commit into
Open
Conversation
MarshalWithExtras applies each extra field with sjson on top of the already-encoded struct, so Go's randomized map iteration order ends up as byte order in the output. The same value marshals to a different byte string on every call, always of the same length. That silently breaks prefix-based prompt caching. On a long chat session our cache hit rate went from ~99% to near zero inside a single tool loop: one assistant message carrying both reasoning_content and reasoning_details re-permuted itself every turn and invalidated the cached prefix from that message onward. 200 marshals of that message produced four distinct 308-byte encodings. The constant length is what makes this hard to catch, since token counts and payload sizes are blind to it. encoding/json has always emitted map keys in sorted order, so sorting here also matches what callers expect from the standard library.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
param.MarshalWithExtrasapplies extra fields to the encoded struct one key at a time with sjson, iterating the map directly. Go randomizes map iteration order, so that order becomes byte order in the request body: the same value serializes to a different byte string on every call, always of the same length.Anything that hashes or diffs a request body loses here, but the way it surfaced for us was prompt caching. Hit rate on a long chat session fell from ~99% to near zero inside a single tool loop, with nothing happening to the conversation other than appending. The cause was one assistant message carrying both
reasoning_contentandreasoning_details: it re-permuted itself every turn and invalidated the provider's cached prefix from that message onward. 200 marshals of that single message produced four distinct 308-byte encodings.The constant length is what makes this expensive to track down. Token counts, message sizes and payload byte counts are all blind to it, so nothing in the usual metrics moves and only a per-message fingerprint shows anything at all.
Changes
extrasin sorted key order inMarshalWithExtras.TestExtraFieldsSorted. Two or more extra fields are required to observe the permutation, since a single key cannot reorder.Sorted order also lines up with
encoding/json, which has always emitted map keys sorted, so structs with extras now behave the way callers expect from the standard library. Structs without extras are unaffected, and there was no stable order to depend on before, so nothing that worked previously changes.Reproduced on v3.39.0, still present on main (52e95a9).