Benchmark-first symbolic retrieval: hard constraints, ablations and curated relevance fixtures

[PhysShell]

Classification

• Change type: enhancement / retrieval evaluation and S7 research infrastructure
• Priority: P2 (deferred)
• Expected value: high before expanding S7 graph traversal or learning similarity weights
• Implementation risk: medium — a weak benchmark can reward the wrong musical similarity and fossilize bad weights
• Recommendation: YES, but only as an offline benchmark slice after S6 acceptance and sufficient curated corpus scale
• Stage ownership: S7 graph layer, with S5 corpus fixtures and later S9 feedback consumption
• Related implementation: core/src/similarity.rs (similarity v3, 15 explainable persisted axes)
• TonalContext status: Phase 1 remains ACCEPTED / CLOSED / FROZEN; this issue must not reopen it
• Production status: research/evaluation only; no generation integration in this issue

Problem

Griff already has a strong first retrieval primitive:

ChunkMeta query
  -> 15 named symbolic similarity axes
  -> versioned WeightPolicy
  -> deterministic explainable ranking

That is substantially better than dropping phrases into an embedding model and trusting cosine distance to understand guitar writing by spiritual osmosis.

What is still missing is evidence that the current ranking retrieves musically useful neighbours for the intended tasks.

Without a benchmark, future work can easily optimize whichever metric is easiest to compute rather than whichever candidates a curator would actually reuse, vary or sequence. The risk increases once S7 adds transition edges, S9 tunes weights, or S15 harmonic facts become available.

Goal

Build an offline, deterministic symbolic retrieval benchmark that answers:

For a given query phrase and retrieval intent,
which corpus candidates should appear in top-K,
which must be excluded by hard constraints,
and which feature axes actually improve relevance?

The benchmark should compare the existing similarity v3 baseline against constrained and ablated policies before adding ANN, learned embeddings or generation integration.

Retrieval intents

Do not collapse all musical similarity into one label. Initial fixture queries should declare an intent such as:

near_variant
rhythmic_analogue
technical_analogue
harmonic_context_match
playability_substitute
contrast_candidate
transition_compatible

Not every intent needs implementation in the first PR. The fixture schema should make intent explicit so a “good neighbour” for rhythm is not incorrectly treated as a universal duplicate.

Query fixture contract

Suggested shape:

RetrievalFixture
  fixture_id
  query_chunk_id
  corpus_digest
  intent
  hard_constraints
  relevant_candidates[]
  graded_candidates[]
  forbidden_candidates[]
  curator_rationale
  fixture_version

Candidate judgments should support grades, for example:

3 = strongly relevant
2 = useful
1 = weakly related
0 = irrelevant
-1 = must not be returned for this intent

Exact scale is reviewable. Judgments and rationale must be data, not hidden in a test method.

Hard constraints

Similarity scoring must remain separate from feasibility.

Depending on available persisted facts, the benchmark should test filters such as:

tuning compatibility
instrument/string-count compatibility
fret/note range
playability ceiling
required/forbidden techniques
meter or bar-length compatibility
minimum measurement/schema version
corpus/source exclusions
optional TonalContext equality/compatibility only when a later accepted contract exposes it

A candidate that violates a hard constraint must not receive salvation through a high weighted average.

Missing facts remain unknown/unsupported, not favorable zero-cost assumptions.

Baselines and ablations

At minimum compare:

1. current uniform similarity v3 over all 15 axes;
2. structure/tag axes only;
3. gesture axes only;
4. complexity axes only;
5. hard-constraint filtering + current v3;
6. one or more intent-specific reviewed policies;
7. deterministic random/order baseline for sanity.

Later experiments may add transition compatibility or tonal facts, but only behind explicit accepted contracts and separate policy versions.

Do not add embeddings merely to create a fashionable baseline. Add them only if the symbolic benchmark exposes a measurable gap that embeddings can plausibly close.

Metrics

Report metrics per intent and overall:

Recall@K
nDCG@K or another graded-ranking metric
MRR where a single best match is meaningful
hard-constraint violation rate
diversity@K / duplicate-neighbour rate
rank stability for identical inputs
coverage: fraction of corpus eligible/measured
curator disagreement where multiple judgments exist

Also report axis/policy explanations for every top-K result so a metric regression can be inspected rather than worshipped.

Benchmark CLI / artifact

A possible interface:

griff retrieval-bench \
  --corpus <manifest> \
  --fixtures <fixtures.jsonl> \
  --policy similarity:v3 \
  --format human|json

Suggested output:

retrieval-bench.json
retrieval-bench.md
per-query/<fixture-id>.json

The JSON artifact should pin:

corpus digest
fixture digest
policy name/version
schema versions
eligible/skipped candidate counts
metric values
top-K candidates with per-axis rationale
constraint decisions

Corpus and provenance discipline

• Query and candidate IDs must resolve against the pinned corpus digest.
• Missing or stale IDs fail loudly.
• Source-song leakage policy must be explicit: some intents may allow same-source variants; others should forbid them.
• Duplicate or near-duplicate chunks should be identified so a top-K full of copies cannot masquerade as quality.
• Curator judgments must preserve who/when/version without requiring personal identity in public fixtures.
• Fixture changes must produce an explained metric delta.

Initial fixture slice

Start small but adversarial:

• 10–20 query chunks across clean/arpeggiated, syncopated, bursty, technical and harmonically distinct material;
• at least one query where current uniform v3 should perform well;
• at least one where tag similarity causes a misleading neighbour;
• at least one where complexity matches but rhythm does not;
• at least one hard-constraint violation that would otherwise rank highly;
• at least one same-song near-duplicate leakage case;
• at least one contrast_candidate case proving that “different but useful” is not ordinary similarity.

The first slice is for validating the benchmark contract, not announcing scientific victory from twelve guitar chunks.

Required controls

1. Identical inputs produce byte-stable rankings and metrics.
2. A forbidden candidate never appears in top-K even with maximum similarity.
3. Unmeasured candidates are reported as coverage exclusions, not scored as zero.
4. Same-source leakage policy changes results explicitly and reproducibly.
5. Removing one axis family produces an inspectable ablation delta.
6. A duplicated candidate set cannot inflate diversity metrics.
7. Fixture/corpus digest mismatch fails.
8. A policy cannot reference unknown axis labels silently.
9. Existing find_similar_chunks semantics remain unchanged unless a separately reviewed policy/filter API is introduced.

Acceptance

• Versioned retrieval fixture schema exists with explicit intent and graded relevance.
• Hard constraints are evaluated separately from similarity scores.
• Existing uniform similarity v3 is recorded as the primary baseline.
• Axis-family and constraint-filter ablations are reported.
• Deterministic Recall@K/graded-ranking, violation, diversity and coverage metrics are emitted.
• Every top-K result retains per-axis rationale and policy provenance.
• Offline fixtures include positive, misleading, forbidden, duplicate and contrast cases.
• Corpus, fixture and policy digests/versions are pinned in artifacts.
• CI can run the small fixture suite without corpus-private files or live services.
• No generation path, feedback learning or TonalContext behavior changes in this issue.

Non-goals

• No ANN/vector database.
• No learned embedding requirement.
• No automatic similarity-weight training.
• No S9 like/dislike integration.
• No S7 DP/Viterbi generation client.
• No automatic scope selection.
• No TonalContext Phase 2 or confidence calibration.
• No claim that one universal similarity ranking serves every musical intent.
• No public release of private corpus material; committed fixtures must be synthetic, minimized or legally safe.

Origin

Inspired by the useful retrieval/reranking separation in Venice's private RAG guide, but adapted to Griff's actual strengths: symbolic features, hard musical constraints, deterministic explanations and curator-labelled relevance. The vector-database furniture is intentionally left outside until measurement proves it belongs in the room.