Skip to content

feat: fixture-backed triage-bug eval - #107

Merged
igmarin merged 1 commit into
mainfrom
eval/triage-bug-fixture
Sep 14, 2026
Merged

igmarin merged 1 commit into
mainfrom
eval/triage-bug-fixture

Conversation

@igmarin

@igmarin igmarin commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Why

ROADMAP item 1: third extract-style eval. triage-bug/basic was four bullets, so the agent never saw a real bug.

What

  • Discount.apply(cents:, percent:) uses cents / 100 * percent (999 cents at 10% → 90, should be 99)
  • Starting test only covers 1000 cents (passes both formulas)
  • Task is a bug report; agent must add a failing test before the fix
  • Condensed skills/triage-bug/SKILL.md
  • Bench test: fixtures exist, bug still present, starting tests pass, skill example ruby -cs

Proof

ruby evals/skills/triage-bug/basic/discount_test.rb
# 1 run, 0 failures

bundle exec ruby -Itest test/evals/triage_bug_basic_test.rb
# 4 runs, 0 failures

bundle exec ruby -Itest test/evaluator/source_path_resolver_test.rb
# 12 runs, 0 failures

Removed leftover Herdr worktree write-yard-docs eval (w2G). This slice lives in triage-bug eval.

Ship Discount.apply with an integer-division bug that 1000-cent tests
miss. The agent must add a 999-cent reproduction, then fix multiply
before divide. Include a condensed skill and a bench test that the bug
is still present.
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: ef7a6a41-c189-4110-a731-adf49dd60749

📥 Commits

Reviewing files that changed from the base of the PR and between 6a4d53f and 27c7557.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • evals/skills/triage-bug/basic/criteria.json
  • evals/skills/triage-bug/basic/discount.rb
  • evals/skills/triage-bug/basic/discount_test.rb
  • evals/skills/triage-bug/basic/task.md
  • skills/triage-bug/SKILL.md
  • test/evals/triage_bug_basic_test.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@igmarin
igmarin merged commit d709344 into main Sep 14, 2026
6 checks passed
@igmarin
igmarin deleted the eval/triage-bug-fixture branch September 14, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant