fix(usage): capture Anthropic cache tokens + emit total_input_tokens (#431)

[initializ-mk]

Fixes #431.

Problem

Forge sets cache_control: ephemeral breakpoints on the Anthropic tools+system prefix, so prompt caching is active by default. Under caching, Anthropic's usage.input_tokens is only the uncached delta — the bulk of the prompt is billed as cache_read_input_tokens (hit) and cache_creation_input_tokens (write). The response parser read only input_tokens/output_tokens, so llm_call events and the run aggregate undercounted real input by orders of magnitude on cache-heavy runs.

Parallel to the SDK fix in initializ-sdk#10.

Fix

1. Parse cache_read_input_tokens / cache_creation_input_tokens in anthropicResponse.Usage; fold them into UsageInfo.TotalTokens (now input + cache_read + cache_creation + output, matching OpenAI's inclusive prompt_tokens).
2. Thread the cache counts through llm.UsageInfo and runtime LLMUsage, each with a TotalInputTokens() helper.
3. Emit on llm_call: cache_read_input_tokens + cache_creation_input_tokens (omitempty) and a summed total_input_tokens (always present). Same field names as initializ-sdk#10, so security-next#36's total_input_tokens ?? input_tokens fallback picks it up unchanged.
4. Aggregate total_input_tokens in usage_accumulator; invocation_complete now carries total_input_tokens_total (+ cache breakdown totals when caching was active).
5. tokens_unavailable now keys off total input, so a fully-cached turn (input_tokens 0, cache_read > 0) is no longer misflagged as a free call.

OpenAI unaffected — its prompt_tokens already includes cached input; the cache fields stay absent.

Orchestrator header

The X-Forge-Tokens-In response header now bills from TotalInputTokens (delta + cache read + cache creation), not the uncached delta — so a cost ceiling-check against the header can't be fooled by a cache-heavy stage. A max(total, input) guard keeps it from ever reporting below the uncached delta if a snapshot lacks the total (mirrors the total_input_tokens ?? input_tokens fallback). Added in a follow-up commit on this branch.

Scope boundary

Streaming (readAnthropicStream) does not emit input tokens at all today; that pre-existing gap is out of scope for this fix.

Tests

• providers: cache extraction + TotalTokens/TotalInputTokens math.
• runtime/audit: breakdown + summed total emitted with exact wire names; total_input_tokens present + cache fields omitted when no caching; cache-read-only turn not flagged unavailable.
• runtime/usage_accumulator: true-input aggregation across a cold+warm pair; cache-read-only run not latched unavailable.

gofmt + golangci-lint clean (0 issues), all affected suites pass.

Acceptance

• A cache-heavy run's summed per-llm_call total_input_tokens matches real input (incl. cache read/creation).
• llm_call carries cache_read_input_tokens + cache_creation_input_tokens + total_input_tokens.
• OpenAI behavior unchanged.

https://claude.ai/code/session_01Hkimw1PDJRY5Dh8BgNQxWJ

[initializ-mk]

Approve-grade — correct root-cause fix, complete end-to-end wiring, thorough tests. Traced the full chain against the branch source.

The fix is end-to-end complete

Under Forge's default cache_control breakpoints, Anthropic's input_tokens is only the uncached delta; the cached prefix bills as cache_read / cache_creation. The parser dropped both. Verified each link:

1. Provider parse (anthropic.go) — parses both cache fields, folds all four into TotalTokens (input+cache_read+cache_creation+output, matching OpenAI's inclusive prompt_tokens). OpenAI / ollama untouched.
2. Types — UsageInfo + LLMUsage both gain the fields and a TotalInputTokens() helper.
3. Emit — total_input_tokens is ALWAYS set (keeps security-next#36's total_input_tokens ?? input_tokens fallback on its fast path); cache fields are nil-pointer-omitempty; tokens_unavailable now keys off totalIn==0 && out==0, so a cache-read-only turn is not misflagged as free.
4. Accumulate — sums cache + true input; snapshot exposes them; the unavailable latch keys off the total.
5. invocation_complete — all THREE emission sites (runner.go 1731 / 2025 / 2323) set total_input_tokens_total + the conditional cache breakdown. N-of-N; no site missed.
6. Header — X-Forge-Tokens-In = max(TotalInputTokens, InputTokens), a defensive guard mirroring the ?? input_tokens fallback.

The critical wiring is clean

The one real risk here is a divergent LLMUsage construction feeding the accumulator WITHOUT the cache fields (which would silently undercount the header + invocation_complete while the audit event looked correct). It is clean: the single AfterLLMCall hook builds usage once with the cache counts and hands the SAME struct to both EmitLLMCall and acc.AddLLMCall. One source, two consumers, no undercount path.

Non-blocking notes

1. Streaming still undercounts. The PR explicitly scopes out readAnthropicStream (emits no input tokens today). The dominant A2A path is non-streaming (the AfterLLMCall hook reads parseAnthropicResponse), so production billing is covered — but a cache-heavy STREAMING invocation stays uncounted until that gap closes. Worth a tracked follow-up.
2. The TotalTokens semantic change also makes the registerLoggingHooks fields["tokens"] value cache-inclusive — intentional and harmless (more accurate; no arithmetic consumer does TotalTokens - InputTokens).

All 10 CI checks green. Field names align with initializ-sdk#10; docs updated in lockstep. Nicely done.

[initializ-mk]

This is the wiring that makes the whole fix hold together, and I verified it is clean. usage is built ONCE here with the cache counts, then the same struct is handed to both EmitLLMCall (so the llm_call event carries total_input_tokens + the cache breakdown) AND acc.AddLLMCall a few lines down (so the accumulator — and thus total_input_tokens_total on invocation_complete + X-Forge-Tokens-In — sums the true input). There is no second LLMUsage construction path that could feed the accumulator the uncached delta while the audit event looked correct, which is exactly the failure mode a change like this invites. One source, two consumers. Nice.

[initializ-mk]

Filed the streaming follow-up the review flagged: #433 — the Anthropic readAnthropicStream path has no message_start case, so it drops input + cache tokens (StreamDelta.Usage carries only output_tokens). Confirmed latent, not live: the A2A executor's ExecuteStream wraps non-streaming Execute→Chat (the #431-fixed parseAnthropicResponse), and the only production ChatStream caller is forge-cli/cmd/ui.go (workspace UI chat), which doesn't feed the audit/billing accumulator. Tracked so the gap closes before any billing path adopts streaming.