docs(readme): clarify framing before hero gif

[smflorentino]

Adds a brief tagline before the hero gif in the README clarifying what coder_eval is (and isn't).

[github-actions]

Claude finished @smflorentino's task in 1m 23s —— View job

---

Review in Progress

I'm reviewing this PR according to the guidelines in .github/code_review.md.

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• View the full diff with git diff origin/main...HEAD
• Read the full README.md file for context
• Analyze cross-file consistency
• Check for "what's missing" items
• Provide structured review feedback

[uipreliga]

Review: coder_eval — pr:16 (1 file) axis:1,2,3,4,5,6,7,8

Scope: pr:16 (1 file) axis:1,2,3,4,5,6,7,8 · branch main (reviewing pr-16 @ 917ea4d) · 917ea4d · 2026-07-13T04:30Z · workflow variant

Change class: trivial — docs-only README framing change: one H1 heading rewrite plus a 5-line positioning paragraph inserted above the hero gif; no code path, schema, or config touched.

coder_eval is in excellent shape — all eight axes are clean of correctness, type-safety, security, resilience, and harness-integrity defects (no confirmed finding can change a task's score or final_status for identical agent output), so the only real risk is positioning/documentation drift on the project's front door, where a newly added "not an agentic coding benchmark" disclaimer contradicts the hero blockquote, the feature bullets, and the PyPI description in the same viewport; bottom line: ship it, and fix the README contradiction as a one-line docs edit.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	10 / 10	0	0	0	0	—
2. Type Safety	10 / 10	0	0	0	0	—
3. Test Health	10 / 10	0	0	0	0	—
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	10 / 10	0	0	0	0	—
7. API Surface & Maintainability	9.5 / 10	0	0	1	0	New "not a benchmark" positioning paragraph contradicts the project's existing benchmark framing (README hero/bullets and pyproject/PyPI description)
8. Evaluation Harness Quality	10 / 10	0	0	0	0	—

Overall Score: 9.9 / 10 · Weakest Axis: API Surface & Maintainability at 9.5 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 0 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 7] New "not a benchmark" positioning paragraph contradicts the project's existing benchmark framing (README hero/bullets and pyproject/PyPI description) (README.md:11) — The added line 11 asserts flatly: Not an "agentic coding" benchmark: it measures how effective your CLI and skills — but seven lines below, the unchanged hero blockquote (lines 18-19) still reads > **The Coding Agents Gym.** A sandboxed, reproducible framework to evaluate, / > benchmark, and A/B-test AI coding agents — Claude Code, Codex, and Google, and the "What you can do with it" list leads with line 33 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted, pass/fail thresholds and line 38 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites. The repo genuinely supports that use case (tasks/, run_command/file_check/reference_comparison criteria, SuiteRollup/threshold gates), so the blanket disclaimer under-sells a real, advertised capability and reads as a self-contradiction on the project's front door — a reader deciding in the first 15 seconds gets "it is not a benchmark" and "benchmark coding agents" in the same viewport. The README already contains the precise, non-contradictory formulation of this exact non-goal at line 162: - **Not a fixed benchmark or leaderboard** — coder_eval scores *your* tasks and ships. Fix: narrow the new sentence to match that existing, accurate phrasing (e.g. Not a fixed agentic-coding leaderboard: it measures how effective your CLI and skills are when used by coding agents.), which keeps the intended SWE-bench differentiation without disclaiming the benchmarking the tool actually does.

Nits

None.

What's Missing

Parallel paths:

• 🟡 Parallel code paths not updated — README.md line 1 was retitled (evaluate & benchmark AI coding agents → evaluate AI coding agents & their skills) and a skills-first/anti-benchmark paragraph added, but the other copies of the same tagline were left on the old framing: pyproject.toml:4 still reads description = "Evaluate, benchmark, and A/B-test AI coding agents (…)" (this is the one-liner PyPI renders directly above the new README body, so the contradiction is visible on the package page itself), and pyproject.toml keywords still lead with benchmark, swe-bench. A positioning change is only real once every surface that carries the positioning string moves together. (trigger: README.md) (restates: Axis 7: New "not a benchmark" positioning paragraph contradicts the project's existing benchmark framing)
• 🟡 Parallel code paths not updated (in-README) — the new paragraph (lines 9-12) sits directly above the unchanged hero blockquote (> **The Coding Agents Gym.** … evaluate, benchmark, and A/B-test AI coding agents, lines 18-21) and the unchanged "What you can do with it" bullets (- **Benchmark coding agents** … line 33; … larger benchmark suites line 38). Editing the H1 + adding a disclaimer without touching the two adjacent blocks that state the opposite is exactly the half-applied-change shape; the accurate non-goal already exists at line 162 (Not a fixed benchmark or leaderboard) and should be the wording reused. (trigger: README.md) (restates: Axis 7: New "not a benchmark" positioning paragraph contradicts the project's existing benchmark framing)
• 🔵 Parallel code paths not updated (product-visible strings) — the same one-line self-description is hard-coded in three shipped places that keep the pre-PR framing and were not touched: src/coder_eval/cli/__init__.py:19 (help="A framework for evaluating AI coding agents") and :44 (same string in the app docstring, i.e. what coder-eval --help prints), src/coder_eval/__init__.py:1, and CLAUDE.md:7. If "…and their skills" is now the positioning, --help is the second-most-read surface after the README and still omits skills entirely. (trigger: README.md)

Display & mapping dicts:

• 🔵 Display asset not extended for the new framing — the PR promotes skills to the headline but the hero asset immediately below it (README.md:9-11) still demonstrates hello_date, a plain write-a-script coding task, with alt text coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task. Neither the GIF nor its alt text shows a skill being evaluated (skill_triggered / a skills suite), so the first visual a reader gets contradicts the sentence directly above it. Either re-record the hero against a skills task or say explicitly in the PR why the asset stays on the coding-task demo. (trigger: README.md)

Downstream consumers:

• 🔵 Downstream consumers of the changed positioning — front doors outside the diff consume this text and are unmentioned in the PR: the GitHub repo About blurb + topics (not versioned in-repo, so it can only be fixed by hand), the PyPI project page (rendered from pyproject.toml description + this README), and the docs entry points (docs/USER_GUIDE.md, docs/tutorials/README.md), none of which mention the skills-first positioning. The PR should state which of these it expects to be updated out-of-band, otherwise the new framing exists on exactly one page. (trigger: README.md)

Tests:

• 🔵 Missing guard for the new coupling — nothing mechanically enforces that README H1/tagline, pyproject.toml:description, and the CLI --help string agree, which is precisely why this PR could move one and leave three behind. A cheap CEnnn lint rule (or a pytest in tests/) asserting the canonical tagline lives in exactly one place — or that the three strings match — would turn this class of drift into a make verify failure rather than a review catch. Note the drift is only mechanically detectable for the literal string copies; the semantic contradiction between "not a benchmark" (line 11) and "Benchmark coding agents" (line 33) needs human judgment and cannot be linted. (trigger: README.md)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] Add CE025 no-unqualified-self-negation as the first text-tier rule (the CE engine in tests/lint/runner.py is ast-only today, so this needs a small TextRule tier: a second registry list ALL_TEXT_RULES run over README.md + docs/**/*.md, with violations flowing into the existing Violation / # noqa machinery). The rule holds a closed table of the capability terms the project affirmatively claims about itself — derived from pyproject.toml:4's description (benchmark, A/B-test, sandboxed, reproducible) plus the README hero blockquote — and flags any line matching Not an?\s+["'*]*(?:[a-z-]+\s+){0,2}<term> unless the negation carries an allowlisted qualifier (fixed, leaderboard, SWE-bench). New file tests/lint/rules/ce025_no_unqualified_self_negation.py, wired into the runner's rule list; runs under the existing make lint / make verify, no new tooling. Prevents: README.md:11 — Not an "agentic coding" benchmark is an unqualified negation of benchmark, a term claimed affirmatively at README.md:18-21, :33, :38 and in pyproject.toml:4. The already-correct phrasing at README.md:162 (Not a fixed benchmark or leaderboard) carries the fixed/leaderboard qualifier and passes, so the rule pins exactly the distinction the reviewer had to draw by hand.
• [ce-lint] Add CE026 positioning-single-source (same text tier): fence the README positioning paragraph with <!-- positioning:start --> / <!-- positioning:end --> and assert pyproject.toml's description is the flattened text of that region (or a declared subset of its capability keywords). The pitch is currently duplicated in three uncoupled places — README hero blockquote (18-21), README capability bullets (33-38), and pyproject.toml:4 which renders as the PyPI landing page — so editing one silently ships an inconsistent front door. Prevents: The grouped half of the README.md:11 finding — the new disclaimer contradicts not only the README hero/bullets but the PyPI description at pyproject.toml:4, a file no author editing README.md would think to open. Makes that coupling mechanical rather than remembered.

Harness improvements (not statically reachable):

• Add a make docs-check target, wired into make verify and .github/workflows/pr-checks.yml, that runs the Markdown gates: the CE text-tier rules above, a relative-link/anchor checker over README.md + docs/** (the README links docs/AB_EXPERIMENTS.md, docs/TASK_DEFINITION_GUIDE.md, docs/tutorials/02-ci-pipeline.md, docs/assets/hero.gif), and a parity assertion that every criterion type in CriterionRegistry appears in the README criteria table. Today verify = format + check + typecheck + test + lint + coverage and not one of those six reads a Markdown file — a docs-only PR passes CI with zero mechanical scrutiny, which is exactly the hole this finding fell through. Why not static: The target is the missing plumbing, not the check: ruff/pyright/CE never see .md files, so even a perfect lint rule has no gate to run in. Link liveness and registry-vs-table parity also need runtime state (filesystem resolution, importing the registry), not a grep. Prevents: README.md:11 — a docs-only change reached human review with no automated gate at all; also prevents the general class of README claims drifting from the code that backs them.
• Extend the review harness's scope classifier so any diff touching README.md, pyproject.toml's description, or docs/** always routes through the API-Surface/Maintainability axis with an explicit front-door coherence prompt: read the changed prose alongside the unchanged hero blockquote, capability bullets, non-goals list, and PyPI description, and flag claims that contradict each other within the same viewport. Why not static: Deciding that two English sentences make opposing claims is semantic judgment. CE025's closed-vocabulary rule catches the narrow Not a <claimed-term> shape only; a reworded disclaimer (e.g. "it isn't really about measuring agents against each other") contradicts the same bullets without matching any pattern. Prevents: README.md:11 and the whole class of positioning contradictions that a line-scoped diff review structurally cannot see — the defect is only visible when the unchanged lines 18-38 are read next to the changed line 11.

Top 5 Priority Actions

1. No scoring/final_status risk was confirmed on any axis — evaluation determinism, criterion aggregation, token/cost accounting, and error-path turn capture all reviewed clean, so nothing needs to be gated on correctness before merge; treat the items below as maintainability polish.
2. Reword /Users/religa/src/coder_eval/README.md:11-12 to reuse the accurate non-goal phrasing already at README.md:162 (e.g. "Not a fixed agentic-coding leaderboard: it measures how effective your CLI and skills are when used by coding agents."), so the disclaimer stops contradicting the hero blockquote (README.md:18-21) and the "Benchmark coding agents" bullet (README.md:33) that a reader sees in the same viewport.
3. Align the PyPI/package description in /Users/religa/src/coder_eval/pyproject.toml:4 ("Evaluate, benchmark, and A/B-test AI coding agents…") with whatever final positioning the README lands on, since it is the second front door and currently reinforces exactly the framing README.md:11 disclaims.
4. Keep the "Known limits & non-goals" section (/Users/religa/src/coder_eval/README.md:162) as the single source of truth for positioning claims and cross-link the intro to it, so future edits to the pitch cannot fork into a second, conflicting statement of the same non-goal.
5. Consider a lightweight docs-consistency check (a CE-style lint or a CI grep) that flags the intro paragraph and the pyproject description diverging from the non-goals section, turning this one-time fix into permanent enforcement per the repo's own "could a lint rule have prevented this?" rule in CLAUDE.md.

---

Stats: 0 🔴 · 0 🟠 · 1 🟡 · 0 🔵 across 8 axes reviewed.

[uipreliga]

🚢