feat: keep skill context out of both judge prompts

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+- `Evaluation::Runner` no longer forwards `skill_context` to either judge. Callers still pass the argument; both prompts get `skill_context: nil`. See `docs/blind-comparisons.md`.
+
 ### Added
 - Fixture-backed `evals/skills/create-service-object/basic`: fat `OrdersController`, passing starting tests, and a condensed `skills/create-service-object/SKILL.md`.
 - Fixture-backed `evals/skills/write-yard-docs/basic`: undocumented `PriceCalculator`, passing starting tests, and a condensed `skills/write-yard-docs/SKILL.md`.

docs/architecture.md

@@ -62,7 +62,7 @@ Exact order for `skill-bench run <eval> --skill <name>`:
 6. `ProviderResolver` builds a `Models::Provider` from `Config` (defaults, then `~/.skill-bench.json`, then `./skill-bench.json`, then `ENV`).
 7. `ContextLoaderService` / `Execution::ContextHydrator` reads skill files. Allowed extensions: `.md`, `.rb`, `.json`, `.yml`, `.yaml`, `.txt`. Per-file cap 50_000 bytes. Total cap 1_000_000 bytes. Symlinks are skipped. Empty context is an error.
 8. `RunnerService` runs baseline and context agents concurrently (`Parallel.map`, two threads). Each agent runs inside `Execution::Sandbox.run`: copy sources into `Dir.mktmpdir`, hardened `git init`, start Docker if available, yield, stop container, delete the tempdir.
-9. `Evaluation::Runner` judges baseline and context concurrently. Baseline judge gets an empty skill context. Context judge gets the XML skill bundle. The judge never sees both outputs in one call.
+9. `Evaluation::Runner` judges baseline and context concurrently. Both judges get `skill_context: nil` (skill text is only for the executing agent). The judge never sees both outputs in one call.
 10. `DeltaReport` computes per-dimension deltas. Verdict is `context_total >= pass_threshold AND total_delta >= minimum_delta`.
 11. `TrendRecorderService` appends `.skill-bench-trends.json` and keeps `.skill-bench-trends.json.bak`.
 12. `CostCalculator` estimates USD from aggregated token usage.

docs/blind-comparisons.md

@@ -0,0 +1,9 @@
+# Comparing skill revisions
+
+Use the same task, fixture commit, acceptance tests, criteria file, model, host configuration, and repetition count for no skills, current skills, and revised skills. Keep a held-out task set. Run the current and revised conditions against the same no-skills baseline protocol; archive the actual outputs and provenance alongside each report.
+
+`Evaluation::Runner` gives both independent judges the same task and criteria. Skill instructions are supplied only to the executing agent. Its `skill_context` argument remains accepted for compatibility but is deliberately omitted from judging. The active evaluator scores one output per independent request, so there is no paired presentation order. A future paired judge must shuffle anonymous outputs and map scores back outside its prompt.
+
+Write criteria as observable task requirements before generating outputs. The legacy `skill_adherence` dimension name remains supported; its description must state an independently assessable requirement rather than ask the judge to infer whether a skill was used. Keep treatment names and instructions out of criteria and agent summaries sent to the judge.
+
+Executable acceptance tests take precedence over prose scores. Preserve test failures and unavailable checks as evidence; a prose score cannot turn either into a passing execution result. Report repetitions, fixture and skill commits, host/model versions, actual usage, and incomplete runs. A small or inconclusive sample establishes no statistical improvement. Paid comparisons require a manual invocation and an explicit usage cap; the unit suite makes no provider calls.

lib/skill_bench/evaluation/runner.rb

@@ -31,6 +31,7 @@ def self.call(task:, criteria:, skill_context:, baseline_output:, context_output
       def initialize(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {})
         @task = task
         @criteria = criteria
+        # Callers still pass skill_context; it is not forwarded to judges.
         @skill_context = skill_context
         @baseline_output = baseline_output
         @context_output = context_output
@@ -66,17 +67,17 @@ def call
       # @return [Array(Hash, Hash)] Baseline and context judge results, in order.
       def run_judges_concurrently
         runs = [
-          -> { judge_run(baseline_output, nil) },
-          -> { judge_run(context_output, skill_context) }
+          -> { judge_run(baseline_output) },
+          -> { judge_run(context_output) }
         ]
         Parallel.map(runs, in_threads: runs.size, &:call)
       end
 
-      def judge_run(output, context)
+      def judge_run(output)
         prompt_result = Judge::Prompt.call(
           task: task,
           criteria: criteria,
-          skill_context: context,
+          skill_context: nil,
           agent_output: output
         )
         return prompt_result unless prompt_result[:success]

test/evaluation_runner_test.rb

@@ -31,7 +31,7 @@ def test_orchestrates_baseline_context_and_judging
       Judge::Prompt.expects(:call).with(
         task: 'Test task',
         criteria: criteria,
-        skill_context: 'Skill context',
+        skill_context: nil,
         agent_output: context_output
       ).returns({ success: true, response: { prompt: 'Context prompt' } })
 
@@ -154,12 +154,32 @@ def test_handles_non_hash_judge_params
       assert result[:success]
     end
 
+    def test_all_conditions_receive_identical_judge_context_without_treatment_instructions
+      prompts = []
+      Judge::Judge.stubs(:call).with do |args|
+        prompts << args[:prompt].gsub(/[0-9a-f]{32}/, 'SENTINEL')
+        true
+      end.returns({ success: true, response: { judge_response: build_judge_response(10, 8, 6, 4, 2) } })
+
+      [nil, 'CURRENT_SKILL_SECRET', 'REVISED_SKILL_SECRET'].each do |context|
+        result = Evaluation::Runner.call(task: 'Solve this task', criteria: build_criteria,
+                                         skill_context: context, baseline_output: 'same output',
+                                         context_output: 'same output')
+
+        assert result[:success]
+      end
+
+      assert_equal 6, prompts.length
+      assert_equal 1, prompts.uniq.length
+      refute_match(/CURRENT_SKILL_SECRET|REVISED_SKILL_SECRET|## Skill Context/, prompts.first)
+    end
+
     private
 
     def stub_prompt_paths
-      Judge::Prompt.stubs(:call).with { |args| args[:skill_context].nil? }
+      Judge::Prompt.stubs(:call).with { |args| args[:agent_output] == 'Baseline diff' }
                                 .returns({ success: true, response: { prompt: 'Baseline prompt' } })
-      Judge::Prompt.stubs(:call).with { |args| args[:skill_context] == 'Skill context' }
+      Judge::Prompt.stubs(:call).with { |args| args[:agent_output] == 'Context diff' }
                                 .returns({ success: true, response: { prompt: 'Context prompt' } })
     end