Retrieval inspector — expose chunks, scores, and match provenance ("chunk viewer")

[calebjpicker]

Feature Description

A read-only inspection surface for the retrieval layer: view any note as the index sees it, view any query as the complete search pipeline answered it, and view a chunk's nearest neighbors in embedding space.

The inspector exposes chunking, vector retrieval, FTS, hybrid fusion, thresholding, and optional reranking without changing any of those behaviors.

Dependencies and Sequencing

This feature spans two retrieval changes currently in flight:

• feat(core): add pluggable semantic vector indexes #1141 — pluggable semantic vector indexes. This reimplements the Milvus work from feat(core): add Milvus semantic vector backend #1041 behind a narrow SemanticVectorIndex boundary. Core owns chunking, embedding generation, and the authoritative SQL chunk manifest; sqlite-vec, pgvector, and external packages such as Milvus own vector persistence and normalized nearest-neighbor lookup. The inspector should target this core-owned manifest and normalized adapter contract, not a backend-specific repository.
• feat(core): add optional cross-encoder reranker stage to search #1143 — optional cross-encoder reranking. This adds a post-retrieval reranker for vector and hybrid search. Query explanations must preserve both the retrieval/fusion rank and the reranker score/rank rather than exposing only the final rewritten score.

Recommended landing order:

1. Land feat(core): add pluggable semantic vector indexes #1141.
2. Rebase and land feat(core): add optional cross-encoder reranker stage to search #1143 on the vector-adapter boundary.
3. Implement this issue against the combined pipeline.

This issue should remain a separate observability feature rather than expanding either dependency PR.

Problem This Feature Solves

BM has two units that never meet in the interface. The note is the authoring unit; search rows and chunks are the retrieval units. A Markdown note first becomes entity, observation, and relation search rows; those rows then become header-aware, individually embedded observation, relation, and roughly 900-character prose chunks.

That two-stage decomposition is uninspectable. Results currently surface a final score and matched text, but not:

• which search row and chunk supplied the match;
• the active vector index, embedding identity, or manifest readiness;
• raw and normalized FTS scores;
• normalized vector similarity;
• threshold or filter rejection reasons;
• the 30% weaker-signal hybrid fusion bonus;
• the pre-rerank rank, reranker score, post-rerank rank, or fallback behavior;
• candidates inside the bounded retrieval window that missed the final page.

Consequences: bad chunking is an invisible cause of bad retrieval; threshold and embedding-model changes can only be evaluated by feel; reranking can improve or worsen an individual result without explaining why; and semantically adjacent wrong claims cannot be inspected as a neighborhood.

Proposed Solution

User workflow

1. Retrieval feels off → bm inspect chunks <note> shows the note's search-row decomposition and the vector chunks produced from each row.
2. A query misses → bm inspect query "<q>" --explain --show-misses shows every ranking stage and the candidates rejected inside the bounded search window.
3. Reranking changes an answer → the same query view shows retrieval rank and score beside reranker rank and score.
4. Auditing a claim → bm inspect neighbors <chunk-id> lists the nearest project-scoped chunks with normalized similarity.

Interface design

CLI/API-first. A GUI client can later render the same responses as hover-for-chunk-plus-score, a per-note chunk map, and a ranking-stage explanation.

Use dedicated diagnostic response schemas rather than adding optional fields to ordinary SearchResult or expanding the MCP search_notes contract.

Technical Approach

1. Chunk inspection

Read the core-owned SQL manifest introduced by #1141 and return:

• owning entity and source search-row type (entity, observation, or relation);
• chunk key and ordinal;
• chunk text and source hash;
• embedding identity/model;
• vector-index identity;
• readiness (pending or ready);
• updated timestamp.

Chunk identifiers are derived projection identifiers and may change after reindexing. Treat them as project-scoped, current-index locators rather than durable note identity.

2. Query execution trace

Capture a typed trace from the same execution that produced the result. Do not rerun FTS, vector retrieval, fusion, or reranking independently to reconstruct an explanation.

For each candidate, preserve as applicable:

• source row and matching chunks;
• raw FTS score and candidate-set-normalized FTS score;
• adapter-normalized vector similarity;
• similarity threshold and filter disposition;
• FTS rank and vector rank;
• fused score and fused rank;
• reranker model, score, and reranked rank;
• final score/rank;
• rejection or fallback reason.

The query-level trace should also include retrieval mode, candidate limit, embedding identity, vector-index identity, fusion formula/version, reranker configuration, and whether a transient reranker failure returned retrieval order.

--show-misses means candidates considered inside the bounded FTS/vector/reranker windows. It must not imply an exhaustive list of every chunk outside those limits.

3. Integration constraints

• feat(core): add pluggable semantic vector indexes #1141 standardizes vector adapters on normalized cosine similarity. The inspector should expose that backend-neutral value, not sqlite-vec or pgvector native distance.
• When feat(core): add optional cross-encoder reranker stage to search #1143 is rebased onto feat(core): add pluggable semantic vector indexes #1141, its retrieval path must consume the normalized similarity directly before reranking.
• feat(core): add optional cross-encoder reranker stage to search #1143 currently replaces the final result score and demotes the un-reranked tail to preserve ordering. The trace must capture retrieval and fusion state before those values are rewritten.
• Use frozen dataclasses for internal trace values and Pydantic models at the API boundary.
• Keep project isolation explicit throughout manifest reads, vector lookup, and result hydration.

4. Neighbor inspection

Implement after chunk and query inspection. #1141 deliberately keeps the external vector-index protocol narrow and supports search by a supplied vector, but not retrieval of a stored vector by key.

Do not expand that new extension contract merely for the first inspector milestone. A follow-up can choose between:

• re-embedding the selected chunk under the active embedding identity and labeling that behavior explicitly; or
• defining an optional backend capability for exact search-by-stored-key.

Implementation Phases

1. Execution trace foundation — typed internal trace that preserves vector, FTS, fusion, threshold, and reranker stages.
2. Chunk and query API/CLI — inspect chunks and inspect query, including bounded misses.
3. Neighbors and GUI affordances — exact neighbor semantics, chunk maps, and hover explanations.

Alternative Solutions

• Status quo — matched text and one final score without provenance.
• Reindex-and-compare by feel — not reproducible and becomes less useful once reranking can rewrite scores.
• External vector-DB tooling — cannot explain BM's search-row decomposition, SQL manifest readiness, FTS, fusion, filters, or reranking.
• Add explanation fields to normal search responses — increases the public contract and payload cost while still encouraging reconstructed rather than execution-native explanations.

Additional Context

Proposed in a March 2026 Discord thread as a "chunk viewer... so we can see the ideas being searched," and endorsed there in the hover-for-chunk-plus-score form.

Confidence/trust weighting (the review-layer direction in #869) is implemented at this scoring layer; the inspector is where those weights would become visible. It also composes with #1154: that proposal instruments the write path, while this issue instruments the read path.

Impact

Retrieval becomes debuggable end to end: users can distinguish bad chunking, stale or pending embeddings, weak vector matches, lexical misses, fusion effects, threshold rejection, and reranker movement.

Threshold tuning, embedding-model swaps, vector-backend changes, and reranker evaluation become evidence-based comparisons. The same substrate can later support regression fixtures and retrieval-quality benchmarks without making diagnostic behavior part of ordinary search.