Skip to content

fix: harden benchmark provider readiness and accounting#2283

Draft
KiDDarn wants to merge 1 commit into
garrytan:mainfrom
KiDDarn:agent/fix-benchmark-provider-adapters
Draft

fix: harden benchmark provider readiness and accounting#2283
KiDDarn wants to merge 1 commit into
garrytan:mainfrom
KiDDarn:agent/fix-benchmark-provider-adapters

Conversation

@KiDDarn

@KiDDarn KiDDarn commented Jul 16, 2026

Copy link
Copy Markdown

What changed

  • prevent nested Codex benchmark runs from inheriting interactive stdin
  • normalize Codex cached input tokens into a separate, discounted bucket
  • show new and cached input separately in table and Markdown reports
  • stop treating stored Gemini OAuth as proof that non-interactive benchmarks are ready
  • classify temporary model-capacity failures separately across Claude, Codex, and Gemini
  • combine subprocess stderr and exception messages without duplicate diagnostics

Why

Nested Codex could wait indefinitely for additional stdin, cached input was double-counted as uncached overhead, and Gemini dry runs could claim readiness for an OAuth path that later fails with UNSUPPORTED_CLIENT. Capacity errors were also falling into the generic unknown bucket because some CLIs report them only in the exception message.

Impact

Benchmark dry runs give more accurate readiness guidance, Codex runs no longer hang on inherited stdin, token/cost reports reflect cache discounts, and temporary provider saturation is actionable instead of opaque. No live model prompts are sent by the added tests.

Validation

  • focused benchmark/provider suite: 41 passed, 0 failed
  • bin/gstack-model-benchmark --dry-run (Codex and Claude ready; stored Gemini OAuth correctly rejected; no prompts sent)
  • npm test (exit 0)
  • git diff --check (pass)
  • gstack redaction scan over added lines and new files: 0 findings

@trunk-io

trunk-io Bot commented Jul 16, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant