fix(judges): a judge config's outputFormat must not reach the provider

[apucacao]

Why

A judge is an ordinary AI Config, so it can carry outputFormat (a JSON Schema). Two code paths read that one field with opposite intent:

• judge_scoring.py ignores it. It appends its formatting instructions and parses the verdict as { score: number 0-1, reasoning: string }. The verdict contract belongs to the SDK.
• The provider handler honors it. Every judge goes through execute_and_track, which hands the whole judge config to the handler. openai-messages sets text.format = json_schema from config.outputFormat; claude and langchain handlers read it too.

So the model is told to produce two different shapes, obeys the provider-level schema, and the judge parser then rejects the verdict. The judge produces no score and tracks no metric, silently.

This is live. A read-only sweep of 215 AI Config variations across four LaunchDarkly connections found 30 with mode: judge, and one of those sets outputFormat to { message: string, script: string }. Nothing about a score. That judge can never return a valid verdict, and nothing warns its author.

Fix

Strip outputFormat from a judge config before it can reach a handler or be stored on a JudgeTask. One helper, without_output_format, lives in judge_scoring.py next to the rest of the verdict contract, and is applied at the four places a judge config is handed off:

• run_judges, before execute_and_track
• build_judge_tasks, before the config is persisted on the task
• run_judge, defensively, so an older serialized task still behaves
• the offline evaluations runner (evaluations/runner.py), when it resolves each judge, so evals.run(...) judges are covered too

When the field was actually present, say so once per judge. The helper returns the same reference when the field is absent and never mutates the input, which may be a cached extract_variation result.

Not in scope, deliberately: the score contract, the parser, the formatting instructions, and handler selection are untouched. Making the SDK honor a judge's schema, or substituting the SDK's own { score, reasoning } schema to get structured output for judges, are both follow-ups.

🤖 Generated with Claude Code

[apucacao]

bugbot run

[apucacao]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 627f1fa. Configure here.}