Skip to content

Cut C: produce and freeze the 56 × 2 blind labels (#456) #508

Description

@pengfei-threemoonslab

Parent: #456. Blocks the v0.16.0 tag.

Cut C produces the corpus's ground truth: the two blind primary labels per case, the owner's adjudication of every disagreement, and the freeze that makes them evidence.

Why this is next

Sourcing is finished. #507 closed out Cut B: all 60 slots pinned, 33 qualifying origins against a floor of 23, and every one of the 28 cells still holding a holdout-eligible slot. Nothing in strata-inventory.csv needs mining before this runs.

Everything downstream — the receipts, the signed artifact, the tag — waits on the labels existing and being frozen.

The protocol is fixed, and it is why an agent cannot self-serve this

Amendment 1: two independent agent sessions on different model families produce the two primary labels; the owner adjudicates every disagreement and is never a rater. Six admissibility conditions apply, including mechanical blindness, a content-addressed archived transcript per label, walked-case disclosure in adjudication, a 5-case calibration round first, and a disclosure block in the published artifact.

Two consequences worth stating plainly, because they are what makes this owner-gated rather than merely unstarted:

  • A session that has read the strata inventory produces inadmissible labels — the inventory names a target decision for every slot.
  • One model family cannot supply both primary labels, so no single assistant can complete this alone.

Clear these four first

From #489, all still open:

  1. Confirm LABELING.md. It was restructured (rater rubric on top, the stripe/ai#232 worked anchor removed because it stated a corpus case's label and the verifier's verdicts, which condition 2 forbids a rater to see). It is the file Amendment 1 names as the rater input, so it needs the owner's sign-off before any label is produced against it.
  2. The Claude harness is unverified — its smoke run failed closed on an expired OAuth token (claude login).
  3. The OpenAI-family harness is unverified — written against codex exec flags from memory, because the local codex npm install has an empty vendor binary directory.
  4. Decide the packet contents. Packets currently hand a rater the head tree plus the diff as "the pinned repository state". Say whether the base tree ships too.

Order of work

  1. Run the calibration round on the five non-corpus cases in calibration.md — four real PRs plus one construction. Which family takes which role is the owner's choice and is recorded with the round.
  2. Land the guide's corrections. This is where the review_required / insufficient_evidence line gets drawn: the miner's needs_human covers both, and 56 labels must not be produced against a guide that has not settled it.
  3. Produce the 56 × 2 labels, blind, with archived transcripts.
  4. Adjudicate every disagreement.
  5. Freeze.

Calibration labels are working material and never corpus labels — the five cases can never fill a slot.

Done when

  • The four preconditions above are cleared and recorded.
  • The calibration round has run and its corrections have landed in LABELING.md, including the review_required / insufficient_evidence rule.
  • 112 labels exist (56 × 2), attributable, with a content-addressed archived transcript per label.
  • Every disagreement carries a third-party adjudication by the owner.
  • Cohen's κ ≥ 0.80 across the two primary labels, computed and recorded.
  • A shipgate.safety_corpus/v4 corpus is emitted whose frozen_labels_sha256 matches the canonical label payload.
  • The Amendment 1 disclosure block is written for the published artifact.

Not in scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Next after P0; blocks other work or ships a misleading resultarea:benchmarkLabeled corpus, accuracy measurementarea:releaseRelease pipeline, packaging, and safety qualification

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions