Cut D: exact-candidate evidence, independent qualification promotion and publication handoff (#456)

[pengfei-threemoonslab]

Current delivery contract — 2026-09-08

This issue owns the shared evidence/promotion machinery and its actual candidate handoff. It does not require a 0.x publication before 1.0. Infrastructure and signer ownership can progress while engineering and beta labeling run.

For 1.0, #512 delivers the frozen human-labeled beta corpus, exact-wheel receipts/index and unsigned qualifying result. This issue validates those inputs, independently signs and promotes them. #512 does not wait for this issue to close. For an optional 0.x track, verify the actual #508 frozen-data handoff before producing its receipts; closed issue state does not prove a freeze, and ignored private files do not prove one is absent.

Current bar

Requirement	pre_1_0 (optional 0.x only)	beta (required for 1.0)
Cases and expected labels	38: 14 passed / 14 review / 10 blocked	80: 30 passed / 20 review / 30 blocked
Strata	21, with four approved single-case blocked cells	21, approved profile weights
Qualifying origins	≥16	≥32
κ / per-stratum holdout	≥0.80 / ≥20%, rounded up	≥0.80 / ≥20%, rounded up
Unsafe auto-passes	0/24 overall and zero per profile	0/50 overall and zero per profile
Safe / blocked exact / review exact	≥13/14; 10/10; 14/14	≥27/30; 30/30; ≥19/20

Read the candidate's typed policy and stdlib restatement, not a hand-derived weighting. Expected-IE has no ground-truth stratum; actual IE loses its applicable exact-outcome score and no-answer attempts remain accounted for. New result envelope is v6; corpus/index v4. Report is currently0.43; #569 must deliberately align the reviewed report1.0 requirement before final v1.0 evidence, without lowering any scoring floor.

Concrete missing prerequisites

Read-only API checks on 2026-09-08 found zero repository Actions variables. The reviewed .github/release-trust-roots.json still has signer_identity: CHANGE_ME; its OIDC issuer is already configured. The independent signer identity and promotion path must be established, not inferred from the availability of URLs or an agent-created GitHub User review. #573 separately owns the actual repository/environment controls; #494 owns release/security/recovery responsibilities.

Remaining acceptance

• Record exact candidate version/tier, source commit, workflow revision, locked build backend, policy bundle and wheel digest. Selected behavior repairs, current-control can return merge authority after the workspace or comparison base changes during the read #567/Qualification: hash and parse the same artifact bytes before scoring release evidence #559, Freeze the report 1.0 contract and prove migration before calling the product v1.0 #569 and A released wheel permanently generates CI pinned to the previous release #570's pre-publication work precede final receipts.
• Validate the applicable corpus's canonical frozen-label digest, primary-label protocol, exposure, per-stratum holdout and adjudication handoff. Beta requires accountable human primary labels.
• Validate reviewed per-case project scope and unique terminal receipts on the exact wheel: real base/head trees, receipt/verify-run/report/verifier/artifact identity joins. Missing, failed, fallback, escaping or mismatched evidence fails by name.
• Consume or produce the v4 receipt index and deterministic v6 result with the corrected Qualification: hash and parse the same artifact bytes before scoring release evidence #559 scorer; the admitted tier has no failures. Beta requires both qualified and production-qualified; pre-1.0 never becomes production-qualified.
• Establish the independent benchmark promotion owner and signing workflow, and commit its exact certificate identity/issuer through a reviewed trust-root change. Authentication must not move into mutable location variables.
• Publish the exact wheel, qualification and Sigstore bundle at content-addressed HTTPS locations, then populate the four existing SAFETY_QUALIFICATION_* repository location variables. Validate the signature and wheel binding against reviewed trust roots.
• Hand those same bytes to Rehearse substantive qualification rejection and the exact final candidate before publication (#456) #510's positive rehearsal. Candidate, workflow, policy or evidence changes invalidate the corresponding freeze/receipts/rehearsal; no restamping.
• Preserve Fixed-history release regression: establish reviewed scope for refused monorepo cases #563/Miner: preserve new and renamed project directories when building the base verifier input #564's cold-start versus reviewed-scope historical views and failed catch-bar obligations. The existing benign-escalation rider still files P0 above15%; a lower rate caused by missing answers is not a gain.
• The release record distinguishes machine qualification from the existing external beta rollout conditions and Run the existing Route H pilot on an installable candidate and decide the v1.0 adoption claim #571's product decision. None substitutes for another.

Sequencing

Promotion infrastructure and independent ownership start now; final signing follows #512's delivered unsigned exact-candidate evidence. #510's negative fixture work can also start now; the positive rehearsal waits for the signed handoff. #570's real tag/PyPI smoke occurs after publication. No dependency requires #509 to close before #512 supplies its inputs.

Historical plan — superseded counts and sequencing; retained for rationale

Parent: #456. Blocks the v0.16.0 tag. Blocked on Cut C (#508) — receipts may only be produced after the labels freeze.

Cut D turns frozen labels into a signed, wheel-bound artifact the release pipeline can verify, and populates the four repository variables that let it find that artifact unattended.

What gets produced, in order

1. One terminal verify receipt per case, run against the release wheel after the freeze. Each needs successful base and head tree-bound runs plus content-addressed verifier_json and report_json artifacts, at report schema 0.43.
2. A shipgate.safety_receipt_index/v4 index binding the exact wheel, corpus, labels, policy bundle, and one shipgate.verification_receipt/v1 plus its shipgate.verify_run/v3 projection per case.
3. safety-qualification.json from scripts/run_safety_qualification.py, which consumes four independently content-addressed inputs (wheel, corpus, receipt index, policy) and emits a deterministic, timestamp-free result: input hashes, strata, per-profile confusion matrices, Wilson 95% intervals, failures[].
4. A signature, from the independent benchmark owner. The runner does not possess or invent signing authority.
5. The four repository variables, populated by that same promotion flow.

The bar it has to clear

Read from pre_release_safety_requirements() on main, not restated by hand:

	Requirement
Unsafe auto-passes	0 — of the 42 cases that should be blocked, reviewed, or abstained on, not one may auto-pass
blocked exact	14 / 14
review_required exact	14 / 14
insufficient_evidence exact	14 / 14
Safe passes	≥ 13 / 14
Cohen's κ	≥ 0.80
Qualifying origins	≥ 23
Holdout per stratum	≥ ceil(2 × 0.20) = 1
Report schema	0.43

Missing, failed, unknown, hash-mismatched, or fallback receipts fail closed; the runner never substitutes a cold-start scan result.

A receipt is rooted at a project, not at a clone

This is the constraint the Cut B close-out surfaced, and it will cost time if it is met case by case at receipt time.

Several candidates sit in repositories where a cold start at the repository root correctly returns refused_unresolved_scope — the workspace holds more than one self-contained project that defines agents, and one manifest describes one agent surface. bytedance/deer-flow and openai/openai-agents-python both do this at their own pinned SHAs. Nine slots sourced before the close-out are in the same position.

The fix is per case, not global: point the run at the project the change is in (for deer-flow, backend/packages/harness), where init --write writes a manifest and the scan reaches a decision. Decide and record each case's scope before starting the receipt run.

The variables, and why the signer identity is not one of them

Four repository-scope variables, currently unset — the repository has zero Actions variables:

SAFETY_QUALIFICATION_WHEEL_URL, SAFETY_QUALIFICATION_WHEEL_FILENAME, SAFETY_QUALIFICATION_JSON_URL, SAFETY_QUALIFICATION_SIGSTORE_BUNDLE_URL.

They carry content-addressed locations only. The Sigstore signer identity and OIDC issuer live in reviewed code at .github/release-trust-roots.json, never in variables: an actor able to set variables could otherwise substitute fabricated evidence and replace the identity that vouches for it, in one step with no diff to review. See docs/distribution.md.

Rider: the benign-escalation re-measurement

Run the cold-start benign-escalation measurement off this same infrastructure. Pre-registered: if the re-measured rate exceeds 15%, a P0 issue is filed — the threshold is fixed before the number exists, deliberately.

The prior figure is benign_escalation_rate 0.286 (4 of 14 safe PRs escalated) from the 2026-W27 re-eval. It predates the first-adoption carve-out and this quarter's scoring work, and docs/engineering/insufficient-evidence-cold-start.md already attributes most of it to apparatus rather than judgement. Measure before concluding anything.

Done when

• Every case's receipt scope is decided and recorded (project root per case, not the clone root).
• One terminal verify receipt per case exists, produced after the freeze, with unique digests.
• A shipgate.safety_receipt_index/v4 index binds wheel, corpus, labels, policy, and receipts.
• safety-qualification.json at envelope shipgate.safety_qualification/v5, tier pre_1_0, qualified: true, production_qualified: false, no failures, bound to the exact release wheel's SHA-256.
• The artifact is signed by the independent benchmark owner and the Sigstore bundle is published.
• All four SAFETY_QUALIFICATION_* repository variables are populated.
• The benign-escalation number is recorded; a P0 is filed if it exceeds 15%.

Current implementation prerequisites — 2026-09-08

The approved #519 policy remains 38 pre-1.0 / 80 beta cases; no threshold was relaxed in this pass. #520's scorer diagnostics are merged in PR #562 and newly produced qualification results use v6 (corpus/receipt-index stay v4; old result envelopes remain readable under their original restrictions). Repair the separately reproduced same-bytes hash/parse gap #559 before final scoring. The #312 fixed-history rerun also fails its unchanged 1.0 release-exit catch bars; #563/#564 must resolve those workflow regressions without relabeling. Freeze the final candidate only after applicable behavior changes, then regenerate its exact receipts and signed qualification. No signed artifact or promotion variable was produced by the implementation PRs.

[pengfei-threemoonslab]

From #520: the bar table in this issue changes. insufficient_evidence exact 14/14 is removed — the value is no longer ground truth and is scored as a coverage miss instead — and "of the 42 cases that should be blocked, reviewed, or abstained on" loses the abstention class. pre_release_safety_requirements() is the source of truth for the new numbers once #520 lands; the sentence "read from pre_release_safety_requirements() on main, not restated by hand" already says the right thing.