Problem
https://helmcode.com/docs/rate-limits tells customers that on a 429 "the Retry-After header tells you how long to wait". All three 429 sites in scripts/litellm-usage-hook/rate_limit_hook.py raise HTTPException(status_code=429, detail=...) with no headers=, so chat completions never send one.
The documentation is not wrong so much as the product is inconsistent: /v1/search does set it (search.go:168). Fixing the hook makes the existing sentence true everywhere, which is better than weakening a correct promise.
Why it matters now
An OpenAI SDK with no Retry-After falls back to its own schedule: openai-python 2.24.0 uses 0.5s x 2^n capped at 8.0s with jitter. Against the rolling budget, which frees up over hours, a client burns all of max_retries in about 15 seconds and surfaces it as our failure.
The first customer who will hit this is about to run load tests, so this stops being cosmetic this week.
The three sites keep the promise to different degrees
Do not write this as one blanket value:
:2091 rolling budget — can be exact. It already computes frees_at.:2101 per-minute TPM — bounded by the window, so ~60s is honest.:2133 concurrency — genuinely cannot predict when a slot frees. A small fixed value in the style of search.go's Retry-After: 1 is the only truthful option.
Follow the convention already set in chat_cap.go:283, which deliberately sets no Retry-After on its 503s precisely because the value could not be kept. A wrong number is worse than none.
Acceptance
A 429 from each of the three sites carries a Retry-After a client can act on, or deliberately carries none with a comment saying why. A test per site pinning which.
Problem
https://helmcode.com/docs/rate-limits tells customers that on a 429 "the
Retry-Afterheader tells you how long to wait". All three 429 sites inscripts/litellm-usage-hook/rate_limit_hook.pyraiseHTTPException(status_code=429, detail=...)with noheaders=, so chat completions never send one.The documentation is not wrong so much as the product is inconsistent:
/v1/searchdoes set it (search.go:168). Fixing the hook makes the existing sentence true everywhere, which is better than weakening a correct promise.Why it matters now
An OpenAI SDK with no
Retry-Afterfalls back to its own schedule: openai-python 2.24.0 uses 0.5s x 2^n capped at 8.0s with jitter. Against the rolling budget, which frees up over hours, a client burns all ofmax_retriesin about 15 seconds and surfaces it as our failure.The first customer who will hit this is about to run load tests, so this stops being cosmetic this week.
The three sites keep the promise to different degrees
Do not write this as one blanket value:
:2091rolling budget — can be exact. It already computesfrees_at.:2101per-minute TPM — bounded by the window, so ~60s is honest.:2133concurrency — genuinely cannot predict when a slot frees. A small fixed value in the style ofsearch.go'sRetry-After: 1is the only truthful option.Follow the convention already set in
chat_cap.go:283, which deliberately sets no Retry-After on its 503s precisely because the value could not be kept. A wrong number is worse than none.Acceptance
A 429 from each of the three sites carries a
Retry-Aftera client can act on, or deliberately carries none with a comment saying why. A test per site pinning which.