Skip to content

Freeze the M0 evaluation bars - #4

Merged
jmjava merged 1 commit into
mainfrom
feat/coder-m0-bar
Oct 7, 2026
Merged

jmjava merged 1 commit into
mainfrom
feat/coder-m0-bar

Conversation

@jmjava

@jmjava jmjava commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • Freeze the general-coding floor: tests, repair, multi-file edit, and explanation. A release fails when any one of those drops more than 5 points below the pinned base model.
  • Freeze the Embabel gain: compile rate, full-task success, hallucinated-API rate, planner success, and repair iterations. The lift counts only when every metric meets its minimum.
  • Freeze the host budget: 7B–14B first. A 32B-class model (30B–34B) waits until docs/runner-stable.md records a runner that fails a known-bad fixture. Downstairs hosts score first. The IDE workstation stays the MCP host.

Test plan

  • python3 scripts/check_m0_bars.py --self-test rejects a quiet edit of the 5-point drop
  • python3 scripts/check_m0_bars.py --against origin/main accepts this initial freeze
  • GitHub Actions m0-bars is green
  • Merge only after that check is green

Made with Cursor

A later edit to the floor, the Embabel lift, or the host budget has to be an amendment, so a bad result cannot quietly move the numbers.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jmjava
jmjava merged commit 2ae654f into main Oct 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant