LiteLLM_DailyTagSpend on the community tenant grows without bound, and the two biggest contributors are strings the client controls.
Measured 2026-09-11
433 MB 636,830 rows 10,915 distinct tags
span: 2026-04-17 → 2026-09-11 (147 days — nothing prunes it)
growth: ~5,000 rows/day, ~400 new distinct tags/day
By rows:
| family |
distinct tags |
rows |
User-Agent:* |
2,878 |
440,578 |
client_ip:* |
7,920 |
96,617 |
country:* |
51 |
82,859 |
cf_colo:* |
64 |
16,724 |
Top single values are User-Agent: Gatus (64,382 rows), User-Agent: Gatus/1.0 (64,373), User-Agent: OpenAI (46,156).
Why it matters
LiteLLM_SpendLogs has 2-day retention and works to the minute. This table has none — 147 days and counting. It is the only unbounded table left in that database, and that database has a history of filling the node disk (285 GB / 84%, July 2026).- Cardinality is client-controlled.
User-Agent and client_ip are both attacker- or accident-driven: a client that varies its UA string per request adds a distinct tag per request. 2,878 distinct UAs today is organic; nothing caps it.
/tag/list already times out — this is the same table behind that.- It is also why the route tag added in nan-devops#333 was checked before shipping: six new values against 10,915 is noise, but the check surfaced this.
Not urgent
433 MB against ~300 GB free. This is "decide before it decides for you", not an incident.
Options, unevaluated
- Retention on
LiteLLM_DailyTagSpend mirroring SpendLogs.
- Stop emitting the unbounded families —
User-Agent and client_ip are the two with no natural ceiling, and neither is read by anything we own as far as I know (unverified).
- Bucket them (UA → client family, IP → /24 or drop).
Found while sizing the route tag in nan-devops#333. Figures are from a single query each, on the community CNPG primary.
LiteLLM_DailyTagSpendon the community tenant grows without bound, and the two biggest contributors are strings the client controls.Measured 2026-09-11
By rows:
User-Agent:*client_ip:*country:*cf_colo:*Top single values are
User-Agent: Gatus(64,382 rows),User-Agent: Gatus/1.0(64,373),User-Agent: OpenAI(46,156).Why it matters
LiteLLM_SpendLogshas 2-day retention and works to the minute. This table has none — 147 days and counting. It is the only unbounded table left in that database, and that database has a history of filling the node disk (285 GB / 84%, July 2026).User-Agentandclient_ipare both attacker- or accident-driven: a client that varies its UA string per request adds a distinct tag per request. 2,878 distinct UAs today is organic; nothing caps it./tag/listalready times out — this is the same table behind that.Not urgent
433 MB against ~300 GB free. This is "decide before it decides for you", not an incident.
Options, unevaluated
LiteLLM_DailyTagSpendmirroring SpendLogs.User-Agentandclient_ipare the two with no natural ceiling, and neither is read by anything we own as far as I know (unverified).Found while sizing the route tag in nan-devops#333. Figures are from a single query each, on the community CNPG primary.