Skip to content

fix(Evals): contains-json scans balanced spans instead of one greedy match (#2189) - #2196

Open
Bowlerjim wants to merge 1 commit into
danielmiessler:mainfrom
Bowlerjim:fix/2189-contains-json-greedy
Open

Bowlerjim wants to merge 1 commit into
danielmiessler:mainfrom
Bowlerjim:fix/2189-contains-json-greedy

Conversation

@Bowlerjim

Copy link
Copy Markdown

Fixes #2189.

/[{[][\s\S]*[}\]]/ is greedy: it runs from the first opening bracket to the last closing bracket anywhere in the output, so any later ] or } (a markdown link, an item[3], a second object) is swallowed into the fragment and JSON.parse fails. A case asserting the agent emitted JSON went red whenever the model also wrote a link, which the runner reports as a real regression.

This scans each opening bracket for its balanced partner, string-aware so a brace inside a JSON string value does not confuse the count, and passes on the first span that parses.

The shipped self-test only covered prefix {"a":1} suffix, which has no trailing bracket, so it passed and hid the bug. Four cases added: JSON followed by a markdown link, JSON followed by item[3], a brace inside a string with a trailing ], and a negative. Self-test 20/20.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PuS77R8a6Lhp7jK6dnwdTh

…match (danielmiessler#2189)

`/[{[][\s\S]*[}\]]/` runs from the first opening bracket to the last closing
bracket anywhere in the output, so any later `]` or `}` — a markdown link, an
`item[3]`, a second object — is swallowed into the fragment and JSON.parse
fails. A case asserting the agent emitted JSON went red whenever the model
also wrote a link, which the runner reports as a real regression.

Scan each opening bracket for its balanced partner (string-aware, so a brace
inside a JSON string value does not confuse the count) and pass on the first
span that parses.

The shipped self-test only covered `prefix {"a":1} suffix`, which has no
trailing bracket, so it passed and hid this. Four cases added: JSON followed
by a markdown link, JSON followed by `item[3]`, a brace inside a string with
a trailing `]`, and a negative (brackets and braces that are not JSON).
Self-test 20/20.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PuS77R8a6Lhp7jK6dnwdTh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals contains-json assertion is greedy: any later bracket swallows the fragment, so output with a markdown link after the JSON fails

1 participant