Follow-up required by the Security review of helmcode/nan-devops#358 (approved on the condition that this issue exists before merge).
The gap
PR #358 gives every fallback group a terminal rung on qwen3.6, so a stream that dies after a failover gets rescued instead of returning 502. That is a clear win for the member, but it moves one accounting case from "metered" to "metered nowhere".
Verified in scripts/litellm-usage-hook/usage_hook.py:
- Before #358. The stream dies.
async_post_call_streaming_iterator_hook never sets finished, so its finally calls _schedule_stream_abort_track(model=_model_name_request(request_data)). That name is the REQUESTED group, which is gated, so the OpenRouter tokens already streamed are posted against the member's cap. The member gets a 502 and is charged for it.
- After #358. The rescue happens inside LiteLLM's stream wrapper, below the proxy hook. The hook sees one continuous stream ending cleanly,
finished becomes True, and the abort meter never fires. async_log_success_event then resolves the model through _model_name_event, gets qwen3.6, which is not in GATED_MODELS, and returns before /track. The partial paid tokens are metered nowhere.
Measured volume on community, 24h to 2026-09-15: 67 requests out of roughly 426,000, i.e. 0.016%, and only the partial tokens before the cut. Enterprise was never separately counted, so its number is unknown rather than small.
Why it was not fixed in #358
The obvious one-liner ("if the served group is un-gated but the requested group is gated, meter against the requested group") would also change long-standing deliberate behaviour: a primary that falls straight to qwen3.6 without burning a single paid token is not metered today, on purpose, because that traffic costs GPU time we already rent. Applying the one-liner would start charging those members, on a volume far larger than 67/day. It would trade a 0.016% visibility gap for over-charging at scale.
Closing this properly needs information the hook does not read today: something that distinguishes "fell to free AFTER burning paid tokens" from "went to free immediately". Candidates are the router's failover trail (previous_models) or the cost on the standard_logging_object.
The part that is a product decision, not an accounting one
Raised by the UX reviewer and worth deciding explicitly rather than inheriting:
Fixing this takes away from the member something #358 gives them. Once the hook can tell the two cases apart, those partial tokens start being charged against the gated cap again, and the member sees consumption for a stream whose model was swapped mid-answer without telling them (the model field changes mid-SSE and nothing else does). Charging for that may well be right, but it should be a deliberate call, not a side effect of the metering fix.
Two sub-questions worth answering at the same time:
- Should the caller be told a fallback served the request at all? A response header (
x-litellm-fallback-served-by or similar) is cheap and turns "the member cannot know" into "the member can know". Today the only signal is the mid-stream model change, which practically no OpenAI-compatible client reads.
- LiteLLM's continuation prompt is hardcoded English ("You are a helpful assistant... Your response should be in continuation of this text:"). On a Spanish-speaking community that can flip the language of an answer mid-sentence.
Also worth folding in (non-blocking findings from the same review)
- The new invariant in
tests/usage-hook/test_litellm_community_usage_hook.py only checks one hop, so a cycle of depth greater than 1 (A: [B] plus B: [A, qwen3.6]) would pass. No current config does this and there is no metering consequence, but walking the chain would close the whole class.
_SELF_RETRY_OK exempts whisper as a target in any position, not just in its own entry. glm5.3: ["whisper"] would skip that check, though the existing "canonicalised or free" invariant still catches it. Anchoring the exemption to the (primary, target) pair would make it explicit.
Related
- helmcode/nan-devops#358 (the change that exposed this)
- The Novita
server overload 429s that pushed the traffic onto the fallback groups in the first place, currently open with the provider.
Follow-up required by the Security review of helmcode/nan-devops#358 (approved on the condition that this issue exists before merge).
The gap
PR #358 gives every fallback group a terminal rung on
qwen3.6, so a stream that dies after a failover gets rescued instead of returning 502. That is a clear win for the member, but it moves one accounting case from "metered" to "metered nowhere".Verified in
scripts/litellm-usage-hook/usage_hook.py:async_post_call_streaming_iterator_hooknever setsfinished, so itsfinallycalls_schedule_stream_abort_track(model=_model_name_request(request_data)). That name is the REQUESTED group, which is gated, so the OpenRouter tokens already streamed are posted against the member's cap. The member gets a 502 and is charged for it.finishedbecomesTrue, and the abort meter never fires.async_log_success_eventthen resolves the model through_model_name_event, getsqwen3.6, which is not inGATED_MODELS, and returns before/track. The partial paid tokens are metered nowhere.Measured volume on community, 24h to 2026-09-15: 67 requests out of roughly 426,000, i.e. 0.016%, and only the partial tokens before the cut. Enterprise was never separately counted, so its number is unknown rather than small.
Why it was not fixed in #358
The obvious one-liner ("if the served group is un-gated but the requested group is gated, meter against the requested group") would also change long-standing deliberate behaviour: a primary that falls straight to
qwen3.6without burning a single paid token is not metered today, on purpose, because that traffic costs GPU time we already rent. Applying the one-liner would start charging those members, on a volume far larger than 67/day. It would trade a 0.016% visibility gap for over-charging at scale.Closing this properly needs information the hook does not read today: something that distinguishes "fell to free AFTER burning paid tokens" from "went to free immediately". Candidates are the router's failover trail (
previous_models) or the cost on thestandard_logging_object.The part that is a product decision, not an accounting one
Raised by the UX reviewer and worth deciding explicitly rather than inheriting:
Fixing this takes away from the member something #358 gives them. Once the hook can tell the two cases apart, those partial tokens start being charged against the gated cap again, and the member sees consumption for a stream whose model was swapped mid-answer without telling them (the
modelfield changes mid-SSE and nothing else does). Charging for that may well be right, but it should be a deliberate call, not a side effect of the metering fix.Two sub-questions worth answering at the same time:
x-litellm-fallback-served-byor similar) is cheap and turns "the member cannot know" into "the member can know". Today the only signal is the mid-streammodelchange, which practically no OpenAI-compatible client reads.Also worth folding in (non-blocking findings from the same review)
tests/usage-hook/test_litellm_community_usage_hook.pyonly checks one hop, so a cycle of depth greater than 1 (A: [B]plusB: [A, qwen3.6]) would pass. No current config does this and there is no metering consequence, but walking the chain would close the whole class._SELF_RETRY_OKexemptswhisperas a target in any position, not just in its own entry.glm5.3: ["whisper"]would skip that check, though the existing "canonicalised or free" invariant still catches it. Anchoring the exemption to the(primary, target)pair would make it explicit.Related
server overload429s that pushed the traffic onto the fallback groups in the first place, currently open with the provider.