Skip to content

feat: keep skill context out of both judge prompts - #113

Merged
igmarin merged 2 commits into
mainfrom
feat/blind-judge-comparisons
Sep 14, 2026
Merged

igmarin merged 2 commits into
mainfrom
feat/blind-judge-comparisons

Conversation

@igmarin

@igmarin igmarin commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Why

feat/blind-judge-comparisons had a dirty tree mixed with leftover docs already on main. The real WIP was never submitted: both judges still received different skill_context on main.

What

  • Evaluation::Runner always calls Judge::Prompt with skill_context: nil
  • Keyword skill_context: stays on the public API
  • Test: three treatment strings produce one identical judge prompt
  • docs/blind-comparisons.md protocol note

Leftover docs (README, AGENTS, help printer, etc.) were discarded; they were stale copies of #100#112.

Proof

bundle exec ruby -Itest test/evaluation_runner_test.rb
# 7 runs, 0 failures

Callers still pass skill_context. Judges receive nil so baseline and
context scoring share the same prompt shape. Document the comparison
protocol in docs/blind-comparisons.md.
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 14 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: 8b515514-020f-4c9f-aa21-2391a9ea1505

📥 Commits

Reviewing files that changed from the base of the PR and between adc56e0 and 0a6d16f.

📒 Files selected for processing (5)
  • CHANGELOG.md
  • docs/architecture.md
  • docs/blind-comparisons.md
  • lib/skill_bench/evaluation/runner.rb
  • test/evaluation_runner_test.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Both judges get skill_context nil; skill XML is agent-only.
@igmarin
igmarin merged commit 810d3db into main Sep 14, 2026
6 checks passed
@igmarin
igmarin deleted the feat/blind-judge-comparisons branch September 14, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant