Land the five rulings and force the insufficient_evidence line (#508)

[pengfei-threemoonslab]

What this PR is

It began as a recovery: PR #514 merged an earlier state of its branch, so main got the packet-integrity work and the codex read-boundary audit but none of the owner's rulings, the calibration record, or the round-2 re-run. This PR carries those onto main — and then carries the whole of Cut C's labelling work that followed from them.

Nine commits, in the order the work actually happened:

commit	what
b46dd48	the five rulings, both calibration re-runs, and the three cases that force the evidence-gap line
c07033a	PR review round: staging erased link changes; equal bytes are not identity; the recorded model was the announced one
ec25527, bcb301b	widen the answer-key check past the one file it knew about; make the answer-free host a script
c6523ce	the first corpus round — 96 labels, and the two blockers that stop the tag
51964a2	guide: a binding is a capability you can name; insufficient_evidence is a signal, not a step
6f2954b	corpus round 2 on the corrected guide — κ = 0.8048 on 47 pairs
39b60e2	the six adjudications, and the third blocker they expose
0c0be62	retire insufficient_evidence as a target, and shorten four cells to the material that exists
cbe2324	second review round: a guide seam that split two raters, and three guards narrower than their own reasons

The owner's six rulings

#	Ruling	Landed
1	Judge the diff. The gate's deterministic deliverable is that capabilities changed, how, and which — not how risky that is. A review_required must hand the person a named capability or it may not block.	LABELING.md rewritten. Engine: #515, #518
2	Agent instruction files are out of scope — semantic; a static gate cannot judge them and would only produce false positives.	Labeled by what else the diff does, else passed. Engine: #516
3	insufficient_evidence only when it names what would resolve it	Superseded by ruling 6. #517 is superseded by #520
4	Identical copies: mechanical, not a convention.	identical_files by sha256 and the executable bit; citations stay authoritative, with a derived evidence_comparison_key beside them (changed by review)
5	The codex read boundary is context management, not sandboxing.	A shell-bearing rater is refused on a host carrying strata-inventory.csv; --working-material for calibration
6	insufficient_evidence is a statement about the reader, not the change. It should be rare, and the reason worked out from first principles.	Retired from the target vocabulary in both tiers. #520

The corpus rounds

round	cases	κ	what it settled
calibration	5 non-corpus	0.4444 → 1.0000 after the rulings	the guide contradicted itself on incomplete guards
round 3	3 constructed	—	forced the IE line nobody had reached in 20 labels
corpus 1	48	0.6111	two blockers: cold-start slots cannot be packeted; the RR↔IE line was mine
corpus 2	47 pairs	0.8048 ✅	IE chosen 18 → 1; disagreements 14 → 6
adjudication	6	—	all six ruled by the owner as third identity, each disclosing a personal walk

47 finals, zero insufficient_evidence — the sixth ruling holds end to end.

The shape change (0c0be62)

Both named policies lose their insufficient_evidence cells: beta 28 strata / 100 cases → 21 / 80, pre_1_0 28 / 56 → 21 / 38. Four pre_1_0 blocked cells hold one case rather than two, because — measured over the 48 blind labels — only four of seven profiles produce a real-world blocked case at all, and each produces exactly one.

No rate moved. Every exact-match floor is still production's rate over the population it governs, rounded up. minimum_qualified_origins becomes 32 / 16 for the same reason: it was approved as 40% of the corpus, and holding the count at 40 over an 80-case corpus would have raised production's demand to half the corpus as a side effect of deleting a decision.

Recorded in docs/release-evidence-policy-decision.md § Amendment 3, STABILITY.md § no corpus case targets insufficient_evidence, and strata-inventory.md § Why four cells hold one case and not two.

Type

• Check or risk-model change
• Input adapter change
• CLI or GitHub Action behavior
• Report, schema, or SARIF output
• Release-policy surface — pre_release_safety_requirements() / production_safety_requirements() and the stdlib restatement in scripts/_release_support.py
• Benchmark harness, calibration cases and documentation

Verification

• pytest -n auto: exit 0. ruff check src scripts tests: clean. CI green on 0c0be62.
• No schema version moves; no field is added, renamed or removed. insufficient_evidence stays in ReleaseDecisionStatus and the verifier still emits it.
• No qualification artifact exists at either tier, so nothing published is invalidated.

Things a reviewer should know

• Labels, transcripts and adjudication material stay uncommitted per calibration.md, on the owner's machine. Ground truth will live only in FrozenSafetyCorpusV1.
• This does not close Cut C: produce and freeze the 56 × 2 blind labels (#456) #508. Cut C's deliverable is a frozen corpus, and three cells are still short (n8n × passed, multi_agent_handoffs × passed, coding_agent_trust_roots × review_required) with one label missing. Those three are the gap rows in strata-inventory.csv.
• One insufficient_evidence is a statement about the reader, not the change: drop it from ground truth, score it as a coverage miss #520 acceptance item is not done here: run_safety_qualification.py still emits insufficient_evidence_exact_rate as a floor metric, which now reads 0/0, passed: true — it should become a per-profile coverage-miss metric instead.
• The rubric changed after the labels were produced (cbe2324). Review found that passed's read-only carve-out and review_required's "a capability you can name" overlapped, and it had already split two raters on one adjudicated case. The guide now states the bounded-read exception in three establishable conditions. Every passed label resting on the carve-out was re-read against the new third condition and none moves, so κ = 0.8048 stands as measured; the correction is for the next round. Written up in corpus-round-2-2026-09-03.md.
• The answer-key check found 8 files in this checkout before this PR's second review round; it finds 52 now — the corpus round records, each construction's CASE.md, and the miner's sweeps (verifier output per pr_url) were all readable by a shell-bearing rater.
• Round 3's cases were constructed by someone who had read the inventory, so like cal-5 they can never be corpus cases.

Refs #508, #515, #516, #518, #520. Supersedes #517.

🤖 Generated with Claude Code

[pengfei-threemoonslab]

I found three integrity issues that should be resolved before this calibration/harness change lands. I reviewed the full origin/main...b46dd4824 diff, reproduced the two packet/evidence failures below, checked the archived calibration artifacts at the location named by the PR, ran pytest tests/test_rater_harness.py -q and Ruff successfully, and ran the repository verifier. Shipgate reports review_publishable / review_required (0 blockers, 28 human-review items for the fixture trust-root files). The broad non-performance suite reached 100%; its only local errors were the packaging fixture's isolated dependency download being denied by this sandbox, while all GitHub CI jobs are green.

[pengfei-threemoonslab]

[P1] Preserve dangling-link changes in constructed diffs

copy_tree_excluding() is used to stage both base and head at line 766, and now drops dangling links before Git sees either state. Returning only the head tree's broken-link list here cannot recover a base-only deletion or a changed link target. I reproduced a constructed case whose base contained removed-link -> missing-target and whose head removed it: the packet had the empty-tree hash for both pins, an empty diff.patch, and no broken_symlinks. That violates this builder's complete base→head-diff contract and can hide the very change under judgment. Please preserve link entries while staging (Git can store the link-target bytes without dereferencing), or otherwise encode both states, and add coverage for base-only and target-change cases.

[pengfei-threemoonslab]

You are right, and I had it backwards — thank you.

The path is evidence. Collapsing repo/runtime/hooks/pre.sh onto repo/a.txt because they hold the same bytes destroys exactly what the citation establishes, and it did it to IndependentHumanLabelV1.evidence_references, which is the authoritative field everything downstream reads. evidence_references_as_cited recorded the damage; it did not repair it.

Taking your second remedy, since it keeps the evidence intact rather than trying to define identity well enough to be safe:

• the rater's citations stay authoritative and untouched on the label;
• the folded form is evidence_comparison_key, derived and recorded beside them, which is all the adjudication case ever needed — two raters citing different copies of identical bytes should not look like a disagreement, and that does not require rewriting either one;
• grouping now keys on content and the executable bit, so your pre.sh / a.txt pair cannot group at all;
• identical_files stays in the manifest as information, and LABELING.md still asks raters to cite one copy — a convention plus a comparison key, with nothing mechanical acting destructively on the guess.

I did not go for the "explicit, validated mirror relationship" option: it would need a definition of mirroring that is itself a guess, and the comparison key gets the benefit without needing one.

[pengfei-threemoonslab]

(Correction: my previous reply on this thread was meant for the identical_file_groups comment — an off-by-one in the script I posted with. The reply to this comment is below; the other threads are being corrected too.)

Reproduced and fixed.

Your fixture exactly: base carrying removed-link -> missing-target, head removing it, produced base_tree == head_tree (bbaca8a6… for both), a 0-byte diff.patch, and no broken_symlinks. A packet asserting nothing changed, for a change whose entire content was that deletion.

The root cause is where the drop happened. copy_tree_excluding stages both trees into the throwaway repository, so whatever it does to a link is the state git records — and it was resolving or dropping links before git saw either side. As you say, git stores a link as its target bytes without dereferencing, so staging now recreates links as links (preserve_symlinks=True), and the drop moved to the packet boundary, which is where a rater's world actually begins.

That also collapsed the broken-link list back to one computation, from the tree about to become repo/, for both external and constructed cases — the per-exporter version existed only because staging had already destroyed the evidence.

Escaping links are no longer refused at staging either, for the same reason: refusing there would block a case whose base carried one and whose head removed it. The packet boundary still refuses them.

Two tests, both failing on the previous commit: base-only deletion (asserts the pins differ and the diff carries deleted file mode 120000) and a changed target (asserts both target strings appear in the diff, and that the link is dropped from repo/ and named in broken_symlinks).

[pengfei-threemoonslab]

[P1] Do not infer semantic identity from equal bytes

This groups every repo file with the same content hash and elects the shortest path as canonical, but a path is part of the evidence: it says which hook/provider/package is loaded and which surface changed. With the current code, a citation to repo/runtime/hooks/pre.sh:1-3 is mechanically rewritten to repo/a.txt:1-3 whenever those files happen to have equal bytes; executable mode and runtime location are ignored. evidence_references_as_cited does not repair the authoritative IndependentHumanLabelV1.evidence_references consumed downstream. Please limit canonicalization to an explicit, validated mirror relationship (including relevant path/mode semantics), or keep original references authoritative and add a separate comparison key rather than collapsing arbitrary duplicates.

[pengfei-threemoonslab]

Confirmed and corrected.

On cal-6.security_governance: the init event says claude-opus-5[1m]; all 14 assistant message events say claude-opus-5. The init value is what the session was configured as and carries decoration the API never returns, so the provenance did not support the attribution the labels claimed.

claude_final now derives the model from the message events. Three details worth flagging:

• it refuses a transcript whose messages name more than one serving model, rather than picking one — if the messages disagree, reviewer_id cannot name the model that ran, so there is no admissible label;
• <synthetic> is discarded (it is what the CLI emits for its own error messages, e.g. the 401 turn), so it cannot become the recorded model;
• the announced value is kept as announced_model with a diagnostic when the two differ, so normalising never silently hides a real disagreement between configuration and service.

All thirteen archived Claude label records across runs, runs-2 and runs-ie2 were re-derived from their own transcripts and now read claude-opus-5, each carrying model_corrected_from: claude-opus-5[1m]. The round record's header is corrected and this is written up as finding 6.

[pengfei-threemoonslab]

(Correction: the reply that landed on this thread a moment ago was meant for the model-identity comment. This is the reply to this one.)

You are right, and I had it backwards — thank you.

The path is evidence. Collapsing repo/runtime/hooks/pre.sh onto repo/a.txt because they happen to hold the same bytes destroys exactly what the citation establishes, and it did so to IndependentHumanLabelV1.evidence_references, the authoritative field everything downstream reads. evidence_references_as_cited recorded the damage; it did not repair it.

Taking your second remedy, because it keeps the evidence intact rather than trying to define identity well enough to be safe:

• the rater's citations stay authoritative and untouched on the label;
• the folded form is evidence_comparison_key, derived and recorded beside them — which is all the adjudication case ever needed, since two raters citing different copies of identical bytes should not look like a disagreement, and that never required rewriting either one;
• grouping now keys on content and the executable bit, so your pre.sh / a.txt pair cannot group at all;
• identical_files stays in the manifest as information, and LABELING.md still asks raters to cite one copy — a convention plus a comparison key, with nothing mechanical acting destructively on the guess.

I did not take the "explicit, validated mirror relationship" option: it needs a definition of mirroring that is itself a guess, and the comparison key gets the benefit without one.

[pengfei-threemoonslab]

[P2] Record the actual Claude model identity

At the archived path this document names, every Claude label in runs, runs-2, and runs-ie2 records model and reviewer_id as claude-opus-5[1m]. That malformed value comes from the system/init event; the same transcripts' assistant message events consistently identify the real model as claude-opus-5. Because condition 3 requires reviewer_id to name the model that ran, the current provenance does not support the claim that these are admissibly attributed labels. Please derive and consistency-check the model from the clean message events (or validate/normalize the init value), then correct the working label records and this round record.

[pengfei-threemoonslab]

Confirmed and corrected.

On cal-6.security_governance: the init event says claude-opus-5[1m]; all 14 assistant message events say claude-opus-5. The init value is what the session was configured as and carries decoration the API never returns, so the provenance did not support the attribution the labels claimed.

claude_final now derives the model from the message events. Three details worth flagging:

• it refuses a transcript whose messages name more than one serving model rather than picking one — if the messages disagree, reviewer_id cannot name the model that ran, so there is no admissible label;
• <synthetic> is discarded (the CLI emits it for its own error turns, e.g. the 401 one), so it can never become the recorded model;
• the announced value is kept as announced_model, with a diagnostic when the two differ, so normalising cannot silently hide a real disagreement between configuration and service.

All thirteen archived Claude label records across runs, runs-2 and runs-ie2 were re-derived from their own transcripts and now read claude-opus-5, each carrying model_corrected_from: claude-opus-5[1m]. The round record's header is corrected, and this is written up as finding 6.

[pengfei-threemoonslab]

Reviewed head 0c0be62d48a8b1def78d65bbc40b12f894ac6258 against base 1942f60b09da9a507b6c2c26a67d7e4bb62e29cf, including the earlier review fixes. I recommend addressing the four P2 findings below before merging: the new decision procedure contradicts the safe-read rule, valid in-packet reads can invalidate a Codex session, calibration-only status is lost from label records, and the answer-key check misses the corpus adjudication records added here.

Validation: all 389 tests in test_rater_harness.py, test_labeling_guide_is_rater_safe.py, test_strata_inventory.py, test_release_pipeline.py, test_safety_qualification.py, and test_safety_qualification_release.py passed. ruff check src scripts tests is clean; self-check reports ready: true; all three GitHub workflows are successful for the reviewed head. I also reproduced the three harness findings with temporary fixtures and injected CLI transcripts, without launching model sessions.

The Python and stdlib release-policy implementations agree on the 38/80-case requirements and the associated rate floors. The static verifier's current control state is review_publishable, with release decision review_required, permissions.update_pr: true, and permissions.merge: false; its protected-surface findings include the added calibration manifests/workflows. This review does not grant merge authority.

The private label/transcript records were outside this code review, so I have not independently re-derived the reported kappa. The PR already discloses the unfinished corpus and the deferred coverage-miss metric; those are not additional findings here.

[pengfei-threemoonslab]

[P2] Keep the decision procedure consistent with the safe-read exception

A new, fully visible read-only lookup within the agent's stated purpose, using existing credentials, is explicitly passed under lines 63–65. It does not match step 1 here, but it introduces a capability the rater can name, so this first-match procedure now requires review_required. The previous procedure retained the qualification that the authority needs human confirmation; this revision removes it, and the revised review_required definition has the same overlap. Two raters can therefore follow different parts of the supplied guide and disagree on an ordinary safe-read case, affecting the labels and kappa. Make the procedure and decision definitions agree on this case before using the guide for another corpus round.

[pengfei-threemoonslab]

Confirmed, and it had already bitten a real case.

You are right that the two readings coexist. What I found checking how far it reached is that they did not stay hypothetical: on mcp_openapi_declared_binding.insufficient_evidence.1 the two raters split on exactly this seam, and the framework_tooling rationale reads almost verbatim from the carve-out — "this is a fully visible, read-only addition within the … server's stated operational scope" — while security_governance reached review_required on "the change registers one genuinely new tool". Two raters, same guide, different halves of it.

Your adjudication is what settles the direction, and it did not settle it the obvious way. The obvious reading of "upheld review_required on a new read-only tool" is a new registered callable is always review_required, which would delete the exception. The upheld rationale says something narrower and better: the new read escapes the bound that constrains the agent's other reads — the per-tool organization allowlist only enforces when a terraform_org_name argument is present, this call takes none, and it returns every organization the token can see including contact addresses. The exception is real; its third condition was missing.

So the guide now names it — the bounded-read exception — in three conditions you can establish from the packet:

1. read-only — no write, no destructive call, no outbound communication;
2. fully visible reach — you can point at the endpoint, path or query; nothing about its extent is decided outside the tree;
3. no new reach — no new credential, permission or scope, and nothing returned that the agent could not already reach under the credential it holds.

Stated in full under passed, referenced from the review_required headline and from step 2, so first-match order is sound wherever a rater enters. Condition 3 is flagged as the one that does the work, and the rule asks for the bound to be cited either way — which is what makes the label checkable by someone who disagrees.

Added a constructed pair that shows where it stops: two tools, both new, both read-only, both fully visible; one reaches a fixed path template under the per-tenant scope that already bounds every other read (passed), one takes no tenant argument so that scope does not apply to it (review_required). Constructed, per the rater-safety guard — neither is a corpus case.

No existing final moves, and I checked rather than assumed. I swept every passed label whose rationale invokes the carve-out (11 of them) and re-read each against condition 3. The only one that adds a new read-only tool reaches a fixed path template under the scope that already bounded the agent, so it stands — and it is now consistent with the adjudicated case rather than in tension with it, which was the thing I most wanted to be true and least expected. The rest are renames, version bumps, behaviour-preserving refactors, or in-tree bindings naming the same tools.

The round is not re-run. κ = 0.8048 was measured under the guide as it stood, and the correction changes no label in it. Recorded in corpus-round-2-2026-09-03.md rather than absorbed quietly, including the part that matters for the next round: the next κ is not comparable to this one on cases where the exception decides.

[pengfei-threemoonslab]

[P2] Exempt paths inside the assigned packet from the parent-directory check

forbidden includes packet.parent, so every canonical absolute path inside the packet contains a forbidden string. I reproduced a complete, valid injected Codex transcript whose only command is cat <packet.resolve()>/repo/agent.py: run_rater() raises RaterError and writes no label, even though the session accessed only its assigned input. The deployment layout also makes REPO_ROOT an ancestor of the packets. Check actual path containment, with the current packet allowed, rather than rejecting any script containing an ancestor's spelling. Add a regression for an absolute in-packet read alongside the existing relative-path examples.

[pengfei-threemoonslab]

Reproduced and fixed. Your fixture exactly: a transcript whose only command is cat <packet.resolve()>/repo/agent.py raised RaterError and wrote no label, for a session that read nothing but its assigned input. And you are right that the deployment layout makes it worse rather than being a corner case — deploy.py puts the packets under the same root REPO_ROOT resolves to, so the checkout root is an ancestor too.

The cause is that the check asked how a path is spelled instead of where it lands. str(root) in script cannot distinguish "inside the packet" from "inside the packet's parent" when one contains the other, and the packet is always inside its parent.

Each token is now resolved against the session's working directory — which is the packet, Invocation.cwd — and judged by containment:

• lands inside the assigned packet → allowed, always;
• lands inside a forbidden root → named (checkout, the round's output directory, the sibling packets);
• otherwise, if the token carried .. → reported as leaving the packet.

Three things fell out that I did not expect and am keeping:

• repo/../repo/agent.py never left the packet, and the old .. token test flagged it. Resolving is what tells the difference between a walk out and a walk that comes back.
• Two spellings of one directory now compare equal. On this host /tmp is a symlink to /private/tmp and the packets live under it, so the substring test could be defeated by spelling the same path the other way. resolve() closes that without being aimed at it.
• --file=<sibling> needed handling. Read whole it is a relative path that lands inside the packet, so a containment test seeing only whole tokens would have passed it — the value after = is now audited too. This one was caught by the old substring test, so it is a regression guard on the new implementation rather than a bug fix, and the test says so.

Tests, all three of the first kind failing on the previous commit (I re-ran the probes against the old implementation to be sure rather than assuming): an absolute in-packet read, an in-packet .., and naming the packet directory itself; plus the flag-value guard; plus a unit-level test that names each side of the containment rule once, keeping the four refusals — relative sibling, absolute sibling, the output directory, the checkout's inventory — and the two allowances a narrow guard has to keep, a bare tool name and /usr/bin/grep.

The docstring now says the check asks where a path lands, and why the obvious spelling test cannot work here.

[pengfei-threemoonslab]

[P2] Persist working-material status independently of host isolation

The new --working-material flag promises that a run's labels are calibration material and never corpus evidence, but the record only carries host_isolation. For Claude that is always no shell; for OpenAI on a clean deployment it is no answer key on host, regardless of the flag. Running both families with working_material=False and True produced otherwise identical records after removing session identities, with the same validated label shape and no calibration marker. A later freeze/audit cannot recover which records the caller explicitly excluded from evidence. Store the boolean or an explicit evidence-eligibility field on every record, even when no host check needed relaxing.

[pengfei-threemoonslab]

Confirmed and fixed. working_material is now on every record, not only where it changed what the host check did.

Your reasoning is the fix's justification, so I have written it into the code comment rather than paraphrasing it: host_isolation cannot carry this. For the claude family it is always "no shell", and on a clean deployment it is "no answer key on host" whichever way the flag was passed — so two runs differing only in this produce identical records once the session identities are removed, and a freeze has nothing to exclude on.

The regression test asserts both halves rather than just the new field: it runs both families under both settings, checks working_material round-trips on all four records, and then asserts the point — that host_isolation is identical for both settings in both families, so it was never the place to read this from.

I kept the field as the caller's declaration (working_material) rather than adding a derived evidence_eligible. The declaration is a fact about how the run was invoked; eligibility is a conclusion the freeze draws, and if a second reason to exclude a record ever appears I would rather add it than have two fields that can disagree about one of them. Happy to switch if you would rather the record state the conclusion.

[pengfei-threemoonslab]

[P2] Include the new corpus-round records in the answer-key check

This PR also adds corpus-round-2026-09-03.md and corpus-round-2-2026-09-03.md; the latter publishes a table mapping six exact case IDs to both primary labels and the owner's final decisions. Neither matches these patterns. With the actual round-2 file copied into an otherwise clean harness root, answer_keys_on_host() returns [] and check_answer_key_not_on_host('openai', working_material=False) returns no answer key on host. Thus a deployment that acquires these records between rounds passes the new pre-launch check despite carrying the answers. Include corpus-round-*.md and cover a root containing only a corpus record, rather than relying on the already-matched inventory files to cause refusal.

[pengfei-threemoonslab]

Confirmed, and the pattern list was narrower than its own stated reason in two more places.

Your case first: the round-2 record publishes case id → both primary labels → the owner's final decision for six cases, and it matched nothing. The part that makes it a real hole rather than an oversight is the one you named — alone on a clean root it passed. Every time I would have noticed, the inventory happened to be sitting beside it and caused the refusal for a different reason.

Chasing that, two classes it never knew about at all:

• benchmark/safety-qualification/constructed/*/CASE.md — each names its target cell and its slot id. The file's own first paragraph says "it names a target decision, so it must never be included in a rater packet", and the check that exists to keep answers off the host did not look for it.
• benchmark/miner/results/*-mined.csv / .jsonl — the sweeps carry head_decision, verify_decision and verify_verdict per pr_url. Only *.labels.csv was matched. This is the worst of the three: it is verifier output, keyed by the exact PRs the inventory then pinned as candidates, which is what condition 2 forbids most directly.

Matched by directory now rather than by naming each file — benchmark/safety-qualification/*.md, constructed/*/CASE.md, benchmark/miner/results/* — because the next round record will be written by someone who will not think to come back and extend a tuple. rater/ is a subdirectory, so the harness itself is not swept up, and deploy.py still produces a clean host (its own assertion covers that). 8 files found in this checkout before, 52 now.

Tests: the enumeration is one list used twice, and the second use is the one you asked for — each class is planted alone on an otherwise empty root and must refuse on its own, so no class can be masked by another.

One thing worth flagging because it is a trap in the other direction: my first draft used a real construction's name in the fixture list, and test_declared_exposure_is_at_least_what_the_tree_shows failed it — naming a construction in tests/ declares it engine-exposed, which would have cost the corpus a holdout-eligible slot. The fixture name is now deliberately not a real construction, with a comment saying why.