From abc4af474ea50666425cf9685f0afdfac966f2c4 Mon Sep 17 00:00:00 2001 From: Ismael Marin Date: Sun, 13 Sep 2026 23:03:55 -0600 Subject: [PATCH 1/2] feat: keep skill context out of both judge prompts Callers still pass skill_context. Judges receive nil so baseline and context scoring share the same prompt shape. Document the comparison protocol in docs/blind-comparisons.md. --- CHANGELOG.md | 3 +++ docs/blind-comparisons.md | 9 +++++++++ lib/skill_bench/evaluation/runner.rb | 9 +++++---- test/evaluation_runner_test.rb | 26 +++++++++++++++++++++++--- 4 files changed, 40 insertions(+), 7 deletions(-) create mode 100644 docs/blind-comparisons.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ae2814..b329c7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed +- `Evaluation::Runner` no longer forwards `skill_context` to either judge. Callers still pass the argument; both prompts get `skill_context: nil`. See `docs/blind-comparisons.md`. + ### Added - Fixture-backed `evals/skills/create-service-object/basic`: fat `OrdersController`, passing starting tests, and a condensed `skills/create-service-object/SKILL.md`. - Fixture-backed `evals/skills/write-yard-docs/basic`: undocumented `PriceCalculator`, passing starting tests, and a condensed `skills/write-yard-docs/SKILL.md`. diff --git a/docs/blind-comparisons.md b/docs/blind-comparisons.md new file mode 100644 index 0000000..ecc6704 --- /dev/null +++ b/docs/blind-comparisons.md @@ -0,0 +1,9 @@ +# Comparing skill revisions + +Use the same task, fixture commit, acceptance tests, criteria file, model, host configuration, and repetition count for no skills, current skills, and revised skills. Keep a held-out task set. Run the current and revised conditions against the same no-skills baseline protocol; archive the actual outputs and provenance alongside each report. + +`Evaluation::Runner` gives both independent judges the same task and criteria. Skill instructions are supplied only to the executing agent. Its `skill_context` argument remains accepted for compatibility but is deliberately omitted from judging. The active evaluator scores one output per independent request, so there is no paired presentation order. A future paired judge must shuffle anonymous outputs and map scores back outside its prompt. + +Write criteria as observable task requirements before generating outputs. The legacy `skill_adherence` dimension name remains supported; its description must state an independently assessable requirement rather than ask the judge to infer whether a skill was used. Keep treatment names and instructions out of criteria and agent summaries sent to the judge. + +Executable acceptance tests take precedence over prose scores. Preserve test failures and unavailable checks as evidence; a prose score cannot turn either into a passing execution result. Report repetitions, fixture and skill commits, host/model versions, actual usage, and incomplete runs. A small or inconclusive sample establishes no statistical improvement. Paid comparisons require a manual invocation and an explicit usage cap; the unit suite makes no provider calls. diff --git a/lib/skill_bench/evaluation/runner.rb b/lib/skill_bench/evaluation/runner.rb index 475958f..03bf81f 100644 --- a/lib/skill_bench/evaluation/runner.rb +++ b/lib/skill_bench/evaluation/runner.rb @@ -31,6 +31,7 @@ def self.call(task:, criteria:, skill_context:, baseline_output:, context_output def initialize(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {}) @task = task @criteria = criteria + # Callers still pass skill_context; it is not forwarded to judges. @skill_context = skill_context @baseline_output = baseline_output @context_output = context_output @@ -66,17 +67,17 @@ def call # @return [Array(Hash, Hash)] Baseline and context judge results, in order. def run_judges_concurrently runs = [ - -> { judge_run(baseline_output, nil) }, - -> { judge_run(context_output, skill_context) } + -> { judge_run(baseline_output) }, + -> { judge_run(context_output) } ] Parallel.map(runs, in_threads: runs.size, &:call) end - def judge_run(output, context) + def judge_run(output) prompt_result = Judge::Prompt.call( task: task, criteria: criteria, - skill_context: context, + skill_context: nil, agent_output: output ) return prompt_result unless prompt_result[:success] diff --git a/test/evaluation_runner_test.rb b/test/evaluation_runner_test.rb index 372b149..05f33ca 100644 --- a/test/evaluation_runner_test.rb +++ b/test/evaluation_runner_test.rb @@ -31,7 +31,7 @@ def test_orchestrates_baseline_context_and_judging Judge::Prompt.expects(:call).with( task: 'Test task', criteria: criteria, - skill_context: 'Skill context', + skill_context: nil, agent_output: context_output ).returns({ success: true, response: { prompt: 'Context prompt' } }) @@ -154,12 +154,32 @@ def test_handles_non_hash_judge_params assert result[:success] end + def test_all_conditions_receive_identical_judge_context_without_treatment_instructions + prompts = [] + Judge::Judge.stubs(:call).with do |args| + prompts << args[:prompt].gsub(/[0-9a-f]{32}/, 'SENTINEL') + true + end.returns({ success: true, response: { judge_response: build_judge_response(10, 8, 6, 4, 2) } }) + + [nil, 'CURRENT_SKILL_SECRET', 'REVISED_SKILL_SECRET'].each do |context| + result = Evaluation::Runner.call(task: 'Solve this task', criteria: build_criteria, + skill_context: context, baseline_output: 'same output', + context_output: 'same output') + + assert result[:success] + end + + assert_equal 6, prompts.length + assert_equal 1, prompts.uniq.length + refute_match(/CURRENT_SKILL_SECRET|REVISED_SKILL_SECRET|## Skill Context/, prompts.first) + end + private def stub_prompt_paths - Judge::Prompt.stubs(:call).with { |args| args[:skill_context].nil? } + Judge::Prompt.stubs(:call).with { |args| args[:agent_output] == 'Baseline diff' } .returns({ success: true, response: { prompt: 'Baseline prompt' } }) - Judge::Prompt.stubs(:call).with { |args| args[:skill_context] == 'Skill context' } + Judge::Prompt.stubs(:call).with { |args| args[:agent_output] == 'Context diff' } .returns({ success: true, response: { prompt: 'Context prompt' } }) end From 0a6d16f93f8e5babc2717fc66032aba3506e1a85 Mon Sep 17 00:00:00 2001 From: Ismael Marin Date: Sun, 13 Sep 2026 23:04:18 -0600 Subject: [PATCH 2/2] docs: architecture matches blind judge prompts Both judges get skill_context nil; skill XML is agent-only. --- docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture.md b/docs/architecture.md index bebe6d6..5932894 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -62,7 +62,7 @@ Exact order for `skill-bench run --skill `: 6. `ProviderResolver` builds a `Models::Provider` from `Config` (defaults, then `~/.skill-bench.json`, then `./skill-bench.json`, then `ENV`). 7. `ContextLoaderService` / `Execution::ContextHydrator` reads skill files. Allowed extensions: `.md`, `.rb`, `.json`, `.yml`, `.yaml`, `.txt`. Per-file cap 50_000 bytes. Total cap 1_000_000 bytes. Symlinks are skipped. Empty context is an error. 8. `RunnerService` runs baseline and context agents concurrently (`Parallel.map`, two threads). Each agent runs inside `Execution::Sandbox.run`: copy sources into `Dir.mktmpdir`, hardened `git init`, start Docker if available, yield, stop container, delete the tempdir. -9. `Evaluation::Runner` judges baseline and context concurrently. Baseline judge gets an empty skill context. Context judge gets the XML skill bundle. The judge never sees both outputs in one call. +9. `Evaluation::Runner` judges baseline and context concurrently. Both judges get `skill_context: nil` (skill text is only for the executing agent). The judge never sees both outputs in one call. 10. `DeltaReport` computes per-dimension deltas. Verdict is `context_total >= pass_threshold AND total_delta >= minimum_delta`. 11. `TrendRecorderService` appends `.skill-bench-trends.json` and keeps `.skill-bench-trends.json.bak`. 12. `CostCalculator` estimates USD from aggregated token usage.