Skip to content

feat: fixture-backed write-yard-docs eval - #106

Merged
igmarin merged 2 commits into
mainfrom
eval/write-yard-docs-fixture
Sep 14, 2026
Merged

igmarin merged 2 commits into
mainfrom
eval/write-yard-docs-fixture

Conversation

@igmarin

@igmarin igmarin commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Why

ROADMAP item 1: only create-service-object had a starting tree. write-yard-docs/basic was four bullets, so baseline and context invented a class and the delta was noise.

What

  • Undocumented PriceCalculator (self.call / #call, raises ArgumentError) plus standalone tests
  • Task names the fixture files and required YARD tags
  • Condensed skills/write-yard-docs/SKILL.md with a valid Ruby example
  • Bench test: eval loads, no @param/@return/@raise yet, fixture tests pass, skill example ruby -cs

Proof

ruby evals/skills/write-yard-docs/basic/price_calculator_test.rb
# 3 runs, 0 failures

bundle exec ruby -Itest test/evals/write_yard_docs_basic_test.rb
# 4 runs, 0 failures

bundle exec rubocop
# 294 files, no offenses

One Herdr worktree (write-yard-docs eval). Did not fan out a second eval — this slice is one fixture.

Ship an undocumented PriceCalculator and passing starting tests so the
agent adds YARD instead of inventing a class. Include a condensed skill
and a bench test that the fixture stays undocumented and green.
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: 4141137a-13bf-4345-92c5-e53e5f2f24b6

📥 Commits

Reviewing files that changed from the base of the PR and between b4c2aae and 6ff32ec.

📒 Files selected for processing (8)
  • CHANGELOG.md
  • evals/skills/write-yard-docs/basic/criteria.json
  • evals/skills/write-yard-docs/basic/price_calculator.rb
  • evals/skills/write-yard-docs/basic/price_calculator_test.rb
  • evals/skills/write-yard-docs/basic/task.md
  • skills/write-yard-docs/SKILL.md
  • test/evals/write_yard_docs_basic_test.rb
  • test/evaluator/source_path_resolver_test.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Shipping skills/write-yard-docs made the fallback tests resolve to the
repo skill instead of the tmpdir skill_sources fixture.
@igmarin
igmarin merged commit 6a4d53f into main Sep 14, 2026
6 checks passed
@igmarin
igmarin deleted the eval/write-yard-docs-fixture branch September 14, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant