bench: publish assistant-run exploratory comparison

[Helweg]

Completed assistant-run comparison

Nine pinned repositories, 100 tasks (54 explicit-symbol, 46 natural-language), seven conditions, 63 cells and 2,100 query-pass records. Self-curated exploratory file discovery, not an independent holdout. No outside invitation; #420 was closed as not planned.

The protocol/task/runtime/interface archive is already tracked in pre-run commit 976ff12; completed results are commit 94636cc.

Condition	Exact-symbol Hit@5	Natural-language Hit@5
OCBI hybrid	40/54 (74.1%)	22/46 (47.8%)
OCBI structural	40/54 (74.1%)	20/46 (43.5%)
CodeGraph	54/54 (100%)	15/46 (32.6%)
codebase-memory	54/54 (100%)	Unsupported selected mapping
grepai	46/54 (85.2%)	33/46 (71.7%)
Literal	43/54 (79.6%)	0/46
Plain-file BM25	44/54 (81.5%)	30/46 (65.2%)

Preserves unfavorable outcomes, all 700 primary rows, paired comparisons, repository-cluster intervals, coverage and read-only post-score diagnoses. No setup/query errors, production changes, scored retries or score amendments.

Verification

Published replay verified 3,936 embedded hashes, rescored all 2,100 records, replayed all 1,962 successful raw returned-path lists, and reproduced the entire primary report exactly using hash-verified pinned source archives. No participant retrieval was rerun. Original private records/archives/indexes retained locally; the 768 MB archive passed gzip integrity and completion-member readback.

No representative-sample, SOTA, coding-productivity or cross-interface speed claim. Different language/ignore coverage, local build bytes, self-curation and Typer/Click ancestry remain explicit limitations. The cohort is now development-exposed.

Landing page: docs/benchmarks/2026-10-06-self-curated-study.md.