Skip to content

Publish five-benchmark routing tables - #29

Merged
SivaKesava1 merged 1 commit into
mainfrom
routing/2026-10-03-five-benchmark
Oct 3, 2026
Merged

SivaKesava1 merged 1 commit into
mainfrom
routing/2026-10-03-five-benchmark

Conversation

@SivaKesava1

Copy link
Copy Markdown
Collaborator

Summary

  • publish six routing tables from the 2026-10-03-five-benchmark training release
  • incorporate accepted Aider Polyglot, SWE-bench Verified, SWE-bench Pro, SWT-Bench Verified, and Terminal-Bench 2 evidence
  • retain schema 5 and all six cost/cost-speed profiles

Training provenance

  • 79,227 observations
  • 21 measured label cells; 210 published cells
  • capability slope: 0.07691434935428793
  • Terminal-Bench 2 contributes one qualifying analyzed cell from 10 tasks in its 73-task fitted common cohort

Replay versus previous tables

Across 1,844 evaluated tasks:

  • economy: +2.33 passes (+0.127 pp), lower credits and runtime
  • balanced: -2.67 passes (-0.145 pp), nearly unchanged credits, faster runtime
  • robust: -0.33 passes (-0.018 pp), lower credits and runtime

Validation

  • final table hashes are byte-identical to the reviewed candidate
  • publication validated all six tables
  • python -m pytest tests/test_routing.py tests/test_package.py: 49 passed, 3 deselected

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SivaKesava1
SivaKesava1 force-pushed the routing/2026-10-03-five-benchmark branch from 47ecccf to 6f6aca6 Compare October 3, 2026 18:08
@SivaKesava1
SivaKesava1 merged commit d2f9102 into main Oct 3, 2026
8 checks passed
@SivaKesava1
SivaKesava1 deleted the routing/2026-10-03-five-benchmark branch October 3, 2026 18:10
@SivaKesava1 SivaKesava1 mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant