You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Anthropic streaming path (readAnthropicStream, forge-core/llm/providers/anthropic.go) never captures input tokens. It handles only the message_delta event, which carries output_tokens:
case"message_delta":
varevanthropicMessageDelta// Usage struct is { OutputTokens int }...ch<- llm.StreamDelta{
FinishReason: finishReason,
Usage: &llm.UsageInfo{ OutputTokens: ev.Usage.OutputTokens },
}
There is no message_start case — and message_start is exactly where Anthropic reports the input usage for a streaming call: message.usage.input_tokens, cache_read_input_tokens, and cache_creation_input_tokens. So on the streaming path:
input_tokens is dropped (not just the cached prefix — the whole input),
cache_read_input_tokens / cache_creation_input_tokens are dropped,
StreamDelta.Usage only ever carries OutputTokens.
This is the streaming sibling of #431 (which fixed the non-streaming parseAnthropicResponse path), plus a pre-existing gap that input tokens were never captured on streaming at all.
Blast radius (why this is latent, not live-billing)
The A2A agent runtime does not stream: the executor's ExecuteStream (forge-core/runtime/loop.go) just wraps Execute, which calls e.client.Chat (non-streaming → parseAnthropicResponse, already fixed in #431). The AfterLLMCall hook that feeds the llm_call audit event + LLMUsageAccumulator + X-Forge-Tokens-In reads that non-streaming response.
The only production caller of the provider ChatStream is forge-cli/cmd/ui.go (the workspace UI chat), which does not feed the audit/billing accumulator. So no live billing path undercounts today.
The risk is regression-latency: the moment any A2A/billing path switches to ChatStream (or a streaming client is wired into the accumulator), input + cache tokens silently read as zero. A cache-heavy streaming invocation would bill as ~free, and tokens_unavailable would even latch true.
Merge the message_start input usage with the message_delta output usage so the final accumulated UsageInfo for the stream is complete (Anthropic splits input at start, output at delta).
Whichever layer accumulates StreamDelta.Usage into the final ChatResponse.Usage must sum input from message_start and output from message_delta — verify a streaming ChatResponse.Usage matches the equivalent non-streaming call.
Acceptance
A streaming Anthropic call's final UsageInfo carries input_tokens + cache_read_input_tokens + cache_creation_input_tokens + output_tokens, matching the non-streaming path for the same prompt.
TotalInputTokens() on a cache-heavy streaming call equals the true input, not zero.
OpenAI streaming unaffected (already emits usage on the final chunk).
Problem
The Anthropic streaming path (
readAnthropicStream,forge-core/llm/providers/anthropic.go) never captures input tokens. It handles only themessage_deltaevent, which carriesoutput_tokens:There is no
message_startcase — andmessage_startis exactly where Anthropic reports the input usage for a streaming call:message.usage.input_tokens,cache_read_input_tokens, andcache_creation_input_tokens. So on the streaming path:input_tokensis dropped (not just the cached prefix — the whole input),cache_read_input_tokens/cache_creation_input_tokensare dropped,StreamDelta.Usageonly ever carriesOutputTokens.This is the streaming sibling of #431 (which fixed the non-streaming
parseAnthropicResponsepath), plus a pre-existing gap that input tokens were never captured on streaming at all.Blast radius (why this is latent, not live-billing)
The A2A agent runtime does not stream: the executor's
ExecuteStream(forge-core/runtime/loop.go) just wrapsExecute, which callse.client.Chat(non-streaming →parseAnthropicResponse, already fixed in #431). TheAfterLLMCallhook that feeds thellm_callaudit event +LLMUsageAccumulator+X-Forge-Tokens-Inreads that non-streaming response.The only production caller of the provider
ChatStreamisforge-cli/cmd/ui.go(the workspace UI chat), which does not feed the audit/billing accumulator. So no live billing path undercounts today.The risk is regression-latency: the moment any A2A/billing path switches to
ChatStream(or a streaming client is wired into the accumulator), input + cache tokens silently read as zero. A cache-heavy streaming invocation would bill as ~free, andtokens_unavailablewould even latch true.Fix
message_startcase toreadAnthropicStreamthat parsesmessage.usage(input_tokens,cache_read_input_tokens,cache_creation_input_tokens) and emits aStreamDeltacarrying them onUsageInfo(reuse theTotalInputTokens()helper + inclusiveTotalTokenssemantics added in Anthropic llm_call undercounts tokens: capture cache_read/creation + emit total_input_tokens (parity with initializ-sdk#10) #431).message_startinput usage with themessage_deltaoutput usage so the final accumulatedUsageInfofor the stream is complete (Anthropic splits input at start, output at delta).StreamDelta.Usageinto the finalChatResponse.Usagemust sum input frommessage_startand output frommessage_delta— verify a streamingChatResponse.Usagematches the equivalent non-streaming call.Acceptance
UsageInfocarriesinput_tokens+cache_read_input_tokens+cache_creation_input_tokens+output_tokens, matching the non-streaming path for the same prompt.TotalInputTokens()on a cache-heavy streaming call equals the true input, not zero.Related
Flagged as a non-blocking follow-up in the #431 (PR #432) review.