Route free-tier users to a cheaper model on the Omni router - #2483
Open
gary149 wants to merge 7 commits into
Open
Route free-tier users to a cheaper model on the Omni router#2483gary149 wants to merge 7 commits into
gary149 wants to merge 7 commits into
Conversation
When LLM_ROUTER_FREE_USER_MODEL is set, users who can't currently pay for inference on the Hub (no PRO subscription, no prepaid credits or valid payment method, no paying billing organization selected) are pinned to that model on the default and agentic routes, with the route's normal candidates as fallbacks. Image requests keep LLM_ROUTER_MULTIMODAL_MODEL since the free model is text-only. Payment status comes from the Hub's OAuth userinfo endpoint (requires the read-billing scope already present in OPENID_SCOPES), cached in-process for 10 minutes per user and failing open to paid so a Hub error never demotes a paying user. The lookup only runs for router-alias requests, after the tier-independent multimodal bypass, and is skipped entirely when the feature is disabled (the default), leaving self-hosted deployments unaffected. Prod and dev pin free users to deepseek-ai/DeepSeek-V4-Flash-0731.
Swap the free-user model from deepseek-ai/DeepSeek-V4-Flash-0731 to zai-org/GLM-5.3-Flash in the prod and dev charts, and add the model to the roster with a description and artifacts support since free users will now see it by default. GLM-5.3-Flash is served first-party by Z.ai, supports tool calling, and is priced well below the previous default route target, so free users' included credits last roughly an order of magnitude longer.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fad14fde30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…seek-routing-7c4d90 # Conflicts: # chart/env/dev.yaml # chart/env/prod.yaml
The multimodal bypass now prefers the free-user model for free users' image requests when the router reports it as multimodal-capable, instead of always using LLM_ROUTER_MULTIMODAL_MODEL. With a text-only free model (the current case for the served GLM-5.3-Flash) nothing changes and the tier is still never resolved for image requests; the switch happens automatically if the served model gains image support.
Concurrent cache misses for the same user each called the rate-limited userinfo endpoint, and a late failure could overwrite a fresh successful entry with the short-lived fail-open one. Track the in-flight lookup per user so parallel requests share a single fetch and a single result.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c89dd4b6b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Free-tier detection calls the Hub's userinfo endpoint with the user's OAuth token. On deployments configured with a third-party OpenID provider that token is not a Hub credential, so the lookup now only runs when the configured provider is huggingface.co (mirroring auth.ts's PROVIDER_URL resolution). With any other provider the feature stays inactive, tokens are never transmitted, and startup validation logs an error for the misconfiguration.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11afb74a72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The multimodal bypass was single-shot: when a free user's image request was served by a multimodal-capable free model, an upstream failure broke the request even though LLM_ROUTER_MULTIMODAL_MODEL was available. The bypass now tries an ordered candidate list (free model first for free users, then the configured multimodal model), matching the fallback behavior of the text routes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When
LLM_ROUTER_FREE_USER_MODELis set, the Omni router pins users who can't currently pay for inference to that model instead of the normal route selection. Prod and dev set it tozai-org/GLM-5.3-Flash(served first-party by Z.ai, tool-capable, already in the model roster since the latest router sync). Leaving the variable empty (the default) disables the feature entirely, so self-hosted deployments see no change.Why
Prod runs with
USE_USER_TOKEN=true, so inference is billed to each user's own HF account. Free users only get a small amount of included monthly credits, and the default route currently sends everyone to Kimi-K2.6, which burns through those credits almost immediately and triggers the "out of credits" modal. At current pricing, an average chat message on GLM-5.3-Flash costs roughly an order of magnitude less than on Kimi-K2.6, taking a free user from ~30 messages a month to a few hundred, while paying users keep the premium routing.How it works
isProorcanPay, or when their selected billing organization can pay (with org billing viaX-HF-Bill-To, the org is the entity charged). Everyone else, including sessions without a user token, is free.read-billingscope is already inOPENID_SCOPES) and cached in-process for 10 minutes per user, with a 1 minute negative cache. Lookups fail open to paid, so a Hub outage never demotes a paying user.[free model, ...route's normal candidates]: if the free model errors, the request falls back to the models already configured inroutes.chat.json. Image requests keepLLM_ROUTER_MULTIMODAL_MODELunless the free model is itself multimodal-capable per the router, in which case it serves free users' images too. The served GLM-5.3-Flash is text-only today, so images stay on the premium multimodal model until Z.ai enables image input, at which point the switch is automatic.default/agentic/multimodal); only the served model changes.read-billingis absent from the scopes.Reviewer notes
src/lib/server/router/endpoint.ts(plain generation) andsrc/lib/server/textGeneration/mcp/routerResolution.ts(MCP flow). The lookup, cache, and validation live in the newsrc/lib/server/router/userTier.ts.npm run checkandnpm run lintclean.