insufficient_evidence is a statement about the reader, not the change: drop it from ground truth, score it as a coverage miss

[pengfei-threemoonslab]

Owner ruling from the first Cut C corpus round (#508), 2026-09-03: insufficient_evidence should be rare, avoided where possible, and the reason worked out from first principles. This is that reasoning, the evidence behind it, and what it changes. It is a child of #518 and it supersedes #517.

Current implementation status — 2026-09-08

#519 has delivered the corrected labeling guide and approved 38-case pre-1.0 / 80-case beta policy. The initial 28-cell/56-case alternatives and calibration counts below remain historical rationale, superseded by that policy.

Delivered engineering:

1. ADK MCP endpoint and credential-reference changes disappear before the capability diff #538 preserves known ADK endpoint/credential-reference changes through evidence and diff surfaces, without claiming remote leaf coverage. FastMCP signature projection must resolve injected Context identity and disclose unsupported schema #539 separately fixes FastMCP signature evidence.
2. PR Qualification: report actual coverage misses without changing safety thresholds #562 reports actual-IE count/rate by profile, denominators, missed cases, named gaps and explicit unscored cases. Expected-IE exactness is explicitly not applicable on the three-label corpus. The shipped metric values and all approved thresholds remain unchanged.
3. ux(reporting): distinguish heuristic action projections from evidence-backed assertions #357 now distinguishes provisional effects from structural evidence and reviewed declarations. Distinguish product defects from user configuration errors in adopter-facing output #328's generator-error slice distinguishes internal generation failure from invalid user configuration.

Remaining:

• Coverage recovery: distinguish reader repair from missing input without inferring ownership from prose #561 must establish whether a coverage remedy is a product reader repair or a supplied input using typed evidence. Current gap kinds and declaration authorship cannot prove that ownership.
• Cut D: exact-candidate evidence, independent qualification promotion and publication handoff (#456) #509/Corpus delivery: current 38-case pre-1.0 and 80-case beta qualification tracks #456 still own the selected corpus, exact candidate receipts and signed qualification. Repair Qualification: hash and parse the same artifact bytes before scoring release evidence #559's artifact read consistency before final scoring.
• The separate benchmark: re-run the labeled corpus on the v0.18 candidate and compare with W27 #312 fixed-history measurement found unresolved workflow regressions Fixed-history release regression: establish reviewed scope for refused monorepo cases #563/Miner: preserve new and renamed project directories when building the base verifier input #564. Passing scorer tests does not satisfy the unchanged historical catch bars.

The epic remains open for these obligations. No authority, binding or deployment claim may be invented to close a coverage gap.

First principles

A verdict routes a change to an outcome. passed ships with nobody looking; review_required puts a named thing in front of a person; blocked does not ship without review. insufficient_evidence routes nowhere. It is an abstention, and an abstention has to earn its place: it is only better than an answer when the answer would be wrong and the abstention leads somewhere.

A runtime or remote binding is itself a nameable capability. This is the mistake the guide and the engine share: they treat "I cannot enumerate the leaves" as "I cannot establish the authority." But the authority is not the leaves. It is the binding:

This agent will call whatever https://mapstools.googleapis.com/mcp advertises, under GOOGLE_MAPS_API_KEY.

That is a complete, citable, decidable statement of authority — and it is a larger one than a fixed tool list, because it is unbounded. Reporting "unbounded" as "unknown" downgrades the strongest finding into an abstention.

What is left for insufficient_evidence is a statement about the reader. Once every binding is nameable, the only honest remaining use is "the evidence I was given is incomplete" — the packet, or the adapter, could not read the input. That is a fact about shipgate, not about the change. It follows that:

insufficient_evidence cannot be a ground-truth label. Ground truth is what a correct gate should do; a correct gate with complete inputs can always name the binding. When shipgate emits it, shipgate is reporting that its own reader failed — a coverage defect, not a release decision.

That is #393 in one sentence: ADK extraction confidence capped at medium made insufficient_evidence every ADK repository's default verdict. An adapter gap was wearing a verdict's clothes.

What the engine actually means by it

insufficient_evidence is emitted when source warnings cross a threshold. The warning's own text (core/source_warnings.py): "this entrypoint's tool surface cannot be enumerated from source, supply it." It is, by construction, a statement about extraction.

Evidence from the corpus round

96 blind labels over 48 cases, two model families, the corrected guide (benchmark/safety-qualification/corpus-round-2026-09-03.md).

κ = 0.6111 against a floor of 0.80, and the disagreement is this line. Codex chose insufficient_evidence 13 times to claude's 5. Collapsing review_required/insufficient_evidence into one class lifts κ to 0.7322. Two raters on different model families, reading the same text over the same evidence, split consistently on one question:

The diff registers a tool by name; its endpoint and credential are citable; its advertised operations live outside the packet. Is that a capability you can name, or a surface that cannot be established?

Both were following the guide, because the guide says both — review_required was rewritten as "adds a capability you can name" (ruling 1) while the insufficient_evidence list still carries "an integration is mounted by name and its capabilities live somewhere the repository does not include."

The four cases where both raters agreed on insufficient_evidence are all nameable. What the tree lacks is a name — the thing is chosen at runtime — and in every one the binding itself is citable:

case	what decides the surface	named as a binding
google_adk.ie.2	application-supplied interceptor objects, registered with an external emitter	arbitrary code in the lifecycle → review_required/blocked
google_adk.ie.3	endpoint from OPENREGISTRY_MCP_URL; tools from the remote tools/list	deployment decides what the agent can call → review_required
multi_agent_handoffs.ie.2	handoffs from TRIAGE_ROUTES, dynamically imported	deployment decides the sub-agents → review_required
n8n.ie.2	a write-capable, unauthorised tool whose target workflow ID comes from the request body	the caller chooses what runs → blocked

The last row is the tell: insufficient_evidence was hiding a block. And its sibling slot, n8n.insufficient_evidence.1, was sourced as insufficient_evidence and both raters labeled it blocked.

The stratum was sourced against the behaviour being doubted. 12 of the 15 insufficient_evidence slots in the inventory carry target_basis: miner_label — chosen with the engine's verdict in view, and the engine over-produces this verdict. A corpus that keeps the stratum enshrines the defect as ground truth, which is the one thing a benchmark must not do.

What changes

1. The rater guide (benchmark/miner/LABELING.md)

The insufficient_evidence section collapses to its honest residue and gains a positive rule:

• Name the binding. When a tool's operations are assembled at runtime, mounted from a remote, or chosen by a deployment value, the capability the agent gains is that binding — cite the call, the endpoint or variable, and the credential. Judge it as any capability: unbounded, unguarded and high-risk → blocked; bounded and attributable → review_required.
• insufficient_evidence is for an incomplete packet, and a rater does not see one. Packets are validated complete before a session starts (MANIFEST.json, broken_symlinks, undescribable_changes). A rater who reaches for the label is being told the guide has a gap; the rationale must say what was unnameable, so the guide can be fixed.

The "mounted by name" bullet, the factory bullet and the environment-variable bullet all move under name the binding.

2. The corpus and its requirements

• insufficient_evidence leaves the target vocabulary. The 28 profile × decision cells become 21; minimum_insufficient_evidence_exact goes; pre_release_safety_requirements() and the beta tier (minimum_insufficient_evidence_exact=19) change together.
• The 15 insufficient_evidence slots in strata-inventory.csv are re-targeted from what the blind raters actually established, or retired. The 48 labeled cases are re-labeled against the corrected guide before any freeze.
• Case count is the owner's arithmetic to settle: 21 × 2 = 42 at the current per-cell count, against floors of 13 passed / 14 review_required / 14 blocked = 41; or 56 kept by raising the per-cell count. The origin floor (23) and the holdout fraction are unchanged either way.

3. Scoring and the product

• An insufficient_evidence verdict on a corpus case scores as a coverage miss. It never matches ground truth, because ground truth no longer contains it. The metric it feeds is adapter coverage, reported per profile, so the misses point at Python FastMCP servers have no route either: @mcp.tool is the largest measured registration idiom #484/The zero-install detector does not read MCP registration sites, so it still reports vendor MCP servers as non-agent projects #485-shaped work rather than at the user.
• The engine keeps the value; it changes what it says. ReleaseDecisionStatus is a shipped enum and is not removed. When emitted, the report frames it as a coverage gap — "shipgate could not enumerate <source>; supply <the named input>" — with An insufficient_evidence verdict is only a finding when it names what would resolve it #517's requirement carried forward: it must name the resolving input, and the route to supply it is the reviewed inventory / RFC: prove the MCP configuration review gap before adding committed capability state #474's committed capability state.

What does not change

• The four-value enum stays; removing a shipped value is a STABILITY.md deprecation and this does not need it.
• No threshold moves. κ stays 0.80; the floors for the three real decisions stay.
• Nothing is adjudicated toward strata-inventory.csv's target_decision.

Shipped surface touched

pre_release_safety_requirements() / production_safety_requirements() (28 → 21 strata, one floor removed); scripts/run_safety_qualification.py (the IE-exact metric becomes a coverage-miss metric); docs/release-evidence-policy-decision.md Amendment 1's corpus shape; the pre_1_0 policy's 56-case statement. Each is a documented change under STABILITY.md; none is a silent one.

Related open issues, and what each needs

issue	relationship	change needed
#517	superseded	close; its "name the resolving input" requirement is carried into §3 above
#518	parent principle	add one sentence: an abstention is not a deliverable — and this issue as a child
#508	Cut C	corpus shape and acceptance change: 21 cells, no IE stratum; plus the 12 shipped-sample slots that have no base//head/ and cannot be a change packet at all (recorded in the round record)
#456	corpus parent	"exactly two in each of the 28 profile × decision strata" → 21; the 56 restated
#509	Cut D	the bar table: drop insufficient_evidence exact 14/14; "42 cases that should be blocked, reviewed, or abstained on" → no abstention class
#510	rehearsal	unchanged in purpose; the "misses the bar" fixture should miss a bar that exists after this
#511, #512	Gate 2, beta corpus	vocabulary only: three ground-truth decisions, not four; beta loses minimum_insufficient_evidence_exact=19
#328	defect vs user error	add IE as the largest instance: an adapter gap surfaced as a verdict the user reads as "your repository is unknowable"
#440	report says what engine did not conclude	add a row: engine concluded "my reader failed"; reader reads "this change is unknowable"
#484, #485	adapter gaps	now the fix route for coverage misses; link back
#474	committed capability state	the user-side route that resolves a coverage miss; link back
#312	miner re-run	unaffected — the three-way miner labels already collapse RR/IE into needs_human; note that ie_rate_on_safe is a coverage-miss rate

Acceptance

• The corrected guide excludes IE from ground truth and names bindings (Land the five rulings and force the insufficient_evidence line (#508) #519).
• Approved pre-1.0/beta requirements and documented corpus-policy migration are implemented (Land the five rulings and force the insufficient_evidence line (#508) #519).
• ADK MCP endpoint and credential-reference changes disappear before the capability diff #538 preserves known ADK remote-connection changes through the existing review/diff surfaces without inventing leaf inventory or effect/authority claims.
• The scorer reports actual-IE count/rate, denominators and cases per profile, with named gaps; expected-IE exactness is explicitly not applicable on the three-label corpus, not presented as successful coverage.
• A fixture proves actual IE still counts against the existing applicable outcome metric; no safety bar or shipped verdict disappears.
• Product gaps and missing user inputs have distinct, truthful recovery routes through existing output surfaces (Distinguish product defects from user configuration errors in adopter-facing output #328/ux(reporting): distinguish heuristic action projections from evidence-backed assertions #357).
• The selected corpus, remaining inventory gaps and signed qualification are resolved through Cut D: exact-candidate evidence, independent qualification promotion and publication handoff (#456) #509/Corpus delivery: current 38-case pre-1.0 and 80-case beta qualification tracks #456 against the approved current policy; this epic does not restate historical candidate arithmetic as the live release bar.

Refs #508, #456, #517, #518, #393, #328, #440, #538, #539.

Implementation update — 2026-09-08

PR #562 delivers the diagnostic scoring slice after two review/address rounds. Qualification v6 records actual-IE per-profile counts, all-case denominators, missed case IDs, existing named gaps and explicit unscored cases; old expected-IE 0/0 is marked not applicable without moving a threshold. The original 38-case safe-pass tolerance remains tested. Older grammar stays readable without fabricated diagnostics. #538 and #357 are also merged.

This epic remains open. Evidence-backed recovery ownership is deferred to #561; selected-corpus/signed qualification remain #509/#456, with the artifact read race #559 to repair before final scoring. The independent fixed-history run #312 found new workflow regressions #563/#564; passing the scorer's tests does not satisfy those release-exit bars.

[pengfei-threemoonslab]

Re-labeled the 48 cases against the corrected guide (benchmark/safety-qualification/corpus-round-2-2026-09-03.md, on the #519 branch).

κ = 0.8048 on 47 complete pairs — over the 0.80 floor. Round 1 on the same 47 cases was 0.6036. insufficient_evidence: 18 labels → 1. The four cases both raters filed there in round 1 are now review_required or blocked from both, naming the binding, which is exactly the movement this issue predicted.

Six disagreements remain (from fourteen); four are on inventory slots sourced as insufficient_evidence, where the raters now split between review_required and passed — a real adjudication question, not a contradiction in the text.

One case has no security_governance label after three independent attempts, each omitting evidence_references; the 48th pair could move κ to either side of 0.80 and is reported as open rather than rounded.

Acceptance items 1 and 2 are met. Items 3–5 (28 → 21 cells, the IE floor, the twelve cold-start slots retired, the IE-sourced slots re-targeted, the STABILITY note) are the next PR.

[pengfei-threemoonslab]

Adjudication outcome: not one of the six landed on insufficient_evidence, including the four on slots the inventory had sourced as exactly that. Across all 47 cases with a final decision, the label appears zero times. The ruling holds end to end.

Acceptance items 1–2 remain met; item 3 (28 → 21 cells, IE floor removed) is deliberately not committed yet — the re-labeled corpus fills only 14 of the 21 cells at two cases each, so writing two-per-cell into pre_release_safety_requirements() today would encode a target known to be unmeetable. The strata shape is settled first (see #508).