Skip to content

feat: show judge variance from trend history - #112

Merged
igmarin merged 1 commit into
mainfrom
feat/judge-run-variance
Sep 14, 2026
Merged

igmarin merged 1 commit into
mainfrom
feat/judge-run-variance

Conversation

@igmarin

@igmarin igmarin commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Why

ROADMAP item 4 remainder: golden traces shipped, but live inter-run totals were not recorded. History already lives in .skill-bench-trends.json.

What

  • TrendCalculator attaches judge_variance (n, mean, stddev, spread) from matching history + current context total
  • Human table prints JUDGE n=… mean=… σ=… spread=… when n ≥ 2
  • No --repeat flag, no verdict-math change

Proof

bundle exec ruby -Itest test/trend_tracker_test.rb
# 10 runs, 0 failures

bundle exec ruby -Itest test/services/delta_table_formatter_test.rb
# 9 runs, 0 failures

Reuse .skill-bench-trends.json for the same eval+skill. Attach sample
mean/stddev/spread of context totals and print JUDGE n/mean/σ/spread.
Verdict math unchanged.
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 40 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: a5e8d2e5-a7b5-40d4-a835-a2a00621da1c

📥 Commits

Reviewing files that changed from the base of the PR and between 2d39e7d and 61fd5cb.

📒 Files selected for processing (6)
  • CHANGELOG.md
  • ROADMAP.md
  • lib/skill_bench/services/delta_table_formatter.rb
  • lib/skill_bench/trend_tracker/trend_calculator.rb
  • test/services/delta_table_formatter_test.rb
  • test/trend_tracker_test.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@igmarin
igmarin merged commit adc56e0 into main Sep 14, 2026
6 checks passed
@igmarin
igmarin deleted the feat/judge-run-variance branch September 14, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant