RH-M0: freeze the external-validity manifest - #69
Conversation
Preregisters the real-history lineage experiment before any B3 run against the STS holdout. No correctness result belongs to this commit. Frozen here: - 63 truth target hashes: 50 natural, 4 rename, 9 copy - baseline definitions B0/B1/B2/B3 - truth protocol: two blinded labels, human adjudication, hash freeze before the first STS B3 run - verdict taxonomy, per axis, no monolithic PASS - 26 input artifact hashes from RH-0, RH-O1 and the targetability census Primary holdout is the 50 natural targets only. The 13 hard-transform targets are reported separately and never enter the primary denominator. The old hard-transform criterion "B3 has fewer unsafe relation errors than B1/B2" is replaced: both baselines require same_pattern_id, pattern_id includes the path, so across a rename or copy they return unresolved, which is not an unsafe error. Their unsafe-error count is naturally zero and "fewer than zero" is unsatisfiable. The criterion is now safe added resolution: zero unsafe errors AND at least three more correct resolutions than the better baseline. merged_symbols has no observation source and none was invented. MERGE-INCONCLUSIVE is the fixed ceiling for this experiment. Step-1 contract at canonical merge 011c136, reviewed head 6dcc02f. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review Scoped final review on exact HEAD |
|
@codex review Final external review requested on exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
requested_head='cde73560f029235959b7775a8466fca059c3f34e'
echo '== HEAD identity =='
git rev-parse HEAD
git cat-file -t "$requested_head"
git show -s --format='commit=%H%nparents=%P%nsubject=%s' "$requested_head"
echo '== Changed files in PR commit range =='
base="$(git merge-base "$requested_head" HEAD^ 2>/dev/null || true)"
git diff-tree --no-commit-id --name-status -r "$requested_head"
echo "parent=${base:-unknown}"
echo '== Commit file statistics and diff =='
git show --stat --oneline --decorate "$requested_head"
git diff "$requested_head^" "$requested_head" -- . ':!*.png' ':!*.pdf'
echo '== Candidate manifest and experiment files =='
fd -HI -t f . | rg -i '(manifest|rh-m0|lineage|external.valid|holdout|target|benchmark|prereg)'
echo '== Text references to frozen identifiers =='
rg -n -i -C 2 \
'b22cb93d7ea1dc2392a318796abc3cfbf67615c300de40dc124f1039b0471d82|011c1362861f6b8b20c45e8ebd5bcb912401c1c0|6dcc02f7d82bebeb5db9be83f77ab8c5455f5692|RH-M0|MERGE-INCONCLUSIVE|NO_TARGET|same_pattern_id' \
.Length of output: 104120 🏁 Script executed: #!/bin/bash
set -euo pipefail
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Manifest checksum =='
sha256sum "$manifest"
cat "$sidecar"
sha256sum --check "$sidecar"
echo '== JSON invariants, samples, and side derivation =='
python3 - <<'PY'
import hashlib, json, subprocess, sys
from collections import Counter, defaultdict
p = "research/rh-m0/rh-m0-manifest.json"
with open(p, encoding="utf-8") as f:
m = json.load(f)
targets = m["frozen_targets"]["targets"]
classes = Counter(t["class"] for t in targets)
print("target_count_declared=", m["frozen_targets"]["count"])
print("target_count_actual=", len(targets))
print("target_classes=", dict(sorted(classes.items())))
print("targets_after_census=", m["sample"]["targets_after_census"])
print("hard_total_declared=", m["hard_challenge"]["frozen_hard_sample"]["total"])
print("no_target_counts=", {k: len(v) for k,v in m["sample"]["no_target"].items() if isinstance(v,list)})
pairs_by_class = defaultdict(set)
for t in targets:
pairs_by_class[t["class"]].add((t["a_sha"],t["b_sha"]))
print("target_pair_counts=", {k:len(v) for k,v in sorted(pairs_by_class.items())})
for cls in ("natural","rename","copy"):
selected=m["sample"]["selected_pairs"][cls]
no_target=len(m["sample"]["no_target"][cls])
targetable=len(pairs_by_class[cls])
print(f"{cls}: selected={selected}, no_target={no_target}, targetable={targetable}, sum={no_target+targetable}")
all_pairs = defaultdict(set)
for cls in ("natural","rename","copy"):
all_pairs[cls] |= pairs_by_class[cls]
all_pairs[cls] |= {(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][cls]}
for a in all_pairs:
for b in all_pairs:
if a < b:
overlap=all_pairs[a] & all_pairs[b]
print(f"pair_overlap_{a}_{b}=",len(overlap))
seed=bytes.fromhex(m["seeds"]["truth_seed"])
bad_sides=[]
for i,t in enumerate(targets):
value=["rh-m0-side/v1", m["seeds"]["truth_seed"], "sts", t["a_sha"], t["b_sha"]]
raw=json.dumps(value, ensure_ascii=False, separators=(",",":")).encode()
expected="A" if (hashlib.sha256(raw).digest()[0]&1)==0 else "B"
if t["side"] != expected: bad_sides.append((i,t["side"],expected))
print("side_rule_mismatches=",bad_sides)
print("unique_target_hashes=",len({t["target_hash"] for t in targets}))
print("malformed_target_hashes=",sum(len(t["target_hash"]) != 64 or any(c not in "0123456789abcdef" for c in t["target_hash"]) for t in targets))
pairs=sorted(set().union(*all_pairs.values()))
bad_objects=[]; bad_adjacency=[]
for a,b in pairs:
for sha in (a,b):
if subprocess.run(["git","cat-file","-e",sha+"^{commit}"]).returncode:
bad_objects.append(sha)
parent=subprocess.check_output(["git","show","-s","--format=%P",b], text=True).split()
if not parent or parent[0] != a:
bad_adjacency.append((a,b,parent[:1]))
print("unique_listed_pairs=",len(pairs))
print("missing_commit_objects=",bad_objects)
print("not_first_parent_adjacent=",bad_adjacency)
PY
echo '== Frozen Step-1 provenance =='
git cat-file -e "${merge}^{commit}"
git cat-file -e "${reviewed}^{commit}"
git show -s --format='merge=%H%nmerge_parents=%P%nmerge_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge"; echo "reviewed_head_is_ancestor_of_merge=$?"
for spec in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
echo "-- $spec"
printf 'merge_blob='
git rev-parse "${merge}:${spec}"
printf 'reviewed_blob='
git rev-parse "${reviewed}:${spec}"
git diff --exit-code "$merge" "$reviewed" -- "$spec"
done
echo '== Normalizer tree identity at merge parents =='
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent_1=' "$path"; git rev-parse "${merge}^1:${path}"
printf '%s parent_2=' "$path"; git rev-parse "${merge}^2:${path}"
printf '%s merge=' "$path"; git rev-parse "${merge}:${path}"
doneLength of output: 4750 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
p="research/rh-m0/rh-m0-manifest.json"
with open(p,encoding="utf-8") as f: m=json.load(f)
targets=m["frozen_targets"]["targets"]
print("json_valid=true")
print("target_count:",m["frozen_targets"]["count"],len(targets),dict(Counter(t["class"] for t in targets)))
print("census_targets:",m["sample"]["targets_after_census"])
print("hard_total:",m["hard_challenge"]["frozen_hard_sample"]["total"])
print("no_target:",{k:len(v) for k,v in m["sample"]["no_target"].items() if isinstance(v,list)})
target_pairs=defaultdict(set)
for t in targets: target_pairs[t["class"]].add((t["a_sha"],t["b_sha"]))
all_pairs=defaultdict(set)
for c in ("natural","rename","copy"):
all_pairs[c]=target_pairs[c]|{(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][c]}
print(f"{c}_accounting:", "selected",m["sample"]["selected_pairs"][c],"targetable",len(target_pairs[c]),"no_target",len(m["sample"]["no_target"][c]))
for a,b in (("natural","rename"),("natural","copy"),("rename","copy")):
print(f"pair_overlap_{a}_{b}:",len(all_pairs[a]&all_pairs[b]))
bad_sides=[]
for i,t in enumerate(targets):
raw=json.dumps(["rh-m0-side/v1",m["seeds"]["truth_seed"],"sts",t["a_sha"],t["b_sha"]],ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0]&1==0 else "B"
if t["side"]!=expected: bad_sides.append(i)
print("side_rule_mismatches:",bad_sides)
print("unique_target_hashes:",len({t["target_hash"] for t in targets}))
print("invalid_target_hashes:",[i for i,t in enumerate(targets) if len(t["target_hash"])!=64 or set(t["target_hash"])-set("0123456789abcdef")])
pairs=set().union(*all_pairs.values())
bad=[]
for a,b in pairs:
subprocess.run(["git","cat-file","-e",a+"^{commit}"],check=True)
subprocess.run(["git","cat-file","-e",b+"^{commit}"],check=True)
first=subprocess.check_output(["git","show","-s","--format=%P",b],text=True).split()[0]
if first!=a: bad.append((a,b,first))
print("listed_pairs:",len(pairs))
print("nonadjacent_or_wrong_direction:",bad)
PY
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Merge provenance =='
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
echo "$path $(git rev-parse "$merge:$path") $(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
echo "$path $(git rev-parse "$merge^1:$path") $(git rev-parse "$merge^2:$path") $(git rev-parse "$merge:$path")"
doneLength of output: 4285 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Sidecar bytes =='
od -An -tx1c research/rh-m0/rh-m0-manifest.sha256
echo '== Side-rule mismatches =='
python3 - <<'PY'
import hashlib,json
with open("research/rh-m0/rh-m0-manifest.json",encoding="utf-8") as f: m=json.load(f)
for i,t in enumerate(m["frozen_targets"]["targets"]):
value=["rh-m0-side/v1",m["seeds"]["truth_seed"],"sts",t["a_sha"],t["b_sha"]]
raw=json.dumps(value,ensure_ascii=False,separators=(",",":")).encode("utf-8")
expected="A" if hashlib.sha256(raw).digest()[0]&1 == 0 else "B"
if t["side"] != expected:
print(json.dumps({
"index":i, "class":t["class"], "a_sha":t["a_sha"], "b_sha":t["b_sha"],
"declared_side":t["side"], "derived_side":expected,
"side_preimage_sha256":hashlib.sha256(raw).hexdigest(),
"target_hash":t["target_hash"]
},separators=(",",":")))
PY
echo '== Step-1 local Git provenance =='
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
if git merge-base --is-ancestor "$reviewed" "$merge"; then echo 'reviewed_head_is_ancestor=true'; else echo 'reviewed_head_is_ancestor=false'; fi
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" \
"$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
doneLength of output: 2118
You are interacting with an AI system. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cde73560f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "<A>", | ||
| "<B>" | ||
| ], | ||
| "side_rule": "sha256(canonical).digest()[0] & 1 ; 0 -> A, 1 -> B", |
There was a problem hiding this comment.
Recomputing the declared canonical side preimage with the frozen truth_seed assigns side B to six hard targets that are stored as side A. For example, the pair 43e13bce.../45eca2c5... hashes to a first byte of 0xa3, so this rule selects B, while its target records A; the same happens for one other rename pair and four copy targets. Because <side> also enters the target-hash preimage, these frozen targets cannot be reproduced from the manifest unless the apparent hard-sample forced-A exception is explicitly specified.
Useful? React with 👍 / 👎.
| "roslyn": "Microsoft.CodeAnalysis.CSharp 4.9.2" | ||
| }, | ||
| "rename_capability": { | ||
| "all_four_targets_labelled": true, |
There was a problem hiding this comment.
At this M0 state the manifest separately says truth_labels_created: false, and the truth protocol requires two blinded labels plus human sign-off, so all_four_targets_labelled: true reads as a contradictory assertion that rename truth labels already exist. If this only means the four targets were selected or tagged with class rename, name the field accordingly so downstream readers cannot treat it as leaked or completed truth.
Useful? React with 👍 / 👎.
| "if_three_or_more_and_any_endpoint_lost_or_invented": "FAIL/PIVOT", | ||
| "if_zero_to_two_genuine_branches": "BRANCH-INCONCLUSIVE regardless of B3 output", | ||
| "validated_requires": [ | ||
| ">= 3 adjudicated truth groups whose truth label is 'branched'", | ||
| "those groups span >= 2 distinct revision pairs", |
There was a problem hiding this comment.
If adjudication finds at least three genuine branches but all belong to one revision pair, this policy assigns no branch verdict: the zero-to-two condition does not apply, the two-pair validation requirement is unmet, and correct endpoint sets do not trigger FAIL/PIVOT. Multiple frozen copy targets share revision pairs, so this is reachable; preregister this case as an explicit verdict, presumably BRANCH-INCONCLUSIVE, before results are observed.
Useful? React with 👍 / 👎.
| "resolved precision >= 95%", | ||
| "zero fabricated new/ended where truth has a counterpart", | ||
| "B3 coverage > B0", | ||
| "report B3-vs-B2 coverage; an absolute loss greater than 10 percentage points requires explicit utility justification and cannot be an unconditional PASS" |
There was a problem hiding this comment.
When B3 loses more than ten percentage points of coverage versus B2, this requirement leaves the natural verdict dependent on an unspecified post-result “utility justification.” It neither defines what makes that justification sufficient nor chooses FAIL versus INCONCLUSIVE, so the same measurements could receive different verdicts after unblinding; freeze an objective disposition or acceptance rule now.
Useful? React with 👍 / 👎.
| [ | ||
| "PASS-NATURAL", | ||
| "PASS-RENAME", | ||
| "PASS-BRANCH", | ||
| "MERGE-INCONCLUSIVE" |
There was a problem hiding this comment.
The first composite example omits the hard axis even though the taxonomy defines five axes and later requires every run to report one verdict per axis; the second example similarly omits rename. Consumers following these frozen examples would produce four-element composites with inconsistent schemas, so both examples should contain natural, rename, hard, branch, and merge verdicts.
Useful? React with 👍 / 👎.
The 63 frozen target hashes are unchanged. Nothing was re-selected. P1, forced-side exception. The manifest described the hash-derived side rule as if it governed every class. It does not: rename and copy candidates must be A-side findings on the transform SOURCE path, so their side is forced to A by construction and the side hash is never consulted. Applying the natural rule to the hard classes yields side B for 6 of the 13 hard targets, and since <side> enters the target-hash preimage those 6 hashes would not reproduce. Confirmed on the frozen data: natural 0 of 50, rename 2 of 4, copy 4 of 9. seeds.side_determination now states the rule per class and carries a reproduction warning with the worked example Codex used. Side now reproduces 63 of 63. P2, truth-label wording. all_four_targets_labelled read as a claim that rename labels already exist, contradicting truth_labels_created=false. Renamed to all_four_targets_enter_labelling. P2, single-pair branch outcome. Three or more genuine branches confined to one revision pair previously fell through every rule. Preregistered as BRANCH-INCONCLUSIVE in an exhaustive four-row outcome table. P2, B2 coverage-loss disposition. "Requires utility justification" left the verdict decidable after unblinding. Frozen: a loss above 10 percentage points caps the natural axis at INCONCLUSIVE, and no after-the-fact justification can raise a capped verdict. P2, composite examples. All examples now carry all five axes. Manifest SHA-256 df0e9f8d7cac50b969ac5d53003885d2ad3d8c05584d011b9e0bcef09ef2c043 supersedes b22cb93d7ea1dc2392a318796abc3cfbf67615c300de40dc124f1039b0471d82. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
Five review findings addressed on a new commitNew exact HEAD The 63 frozen target hashes are byte-identical between the two commits. Nothing was re-selected.
On P1. The finding is correct and the example reproduces exactly. Recomputing the natural side rule over the frozen On the B2 disposition. The previous wording left the verdict decidable after unblinding. It is now frozen: a coverage loss above 10 percentage points caps the natural axis at Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='9c0a8618f131e1bf0362d80538bb2af41f2d2c9a'
parent='cde73560f029235959b7775a8466fca059c3f34e'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Exact commit =='
test "$(git rev-parse HEAD)" = "$head"
git show -s --format='commit=%H%nparents=%P%nsubject=%s' "$head"
test "$(git rev-parse "$head^")" = "$parent"
git diff-tree --no-commit-id --name-status -r "$head"
echo '== Changed manifest diff =='
git diff --no-ext-diff --unified=80 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Checksum =='
sha256sum "$manifest"
cat "$sidecar"
sha256sum --check "$sidecar"
echo '== Manifest invariants and preservation =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
oldp = "research/rh-m0/rh-m0-manifest.json"
new = json.loads(subprocess.check_output(
["git", "show", "cde73560f029235959b7775a8466fca059c3f34e:" + oldp], text=True))
with open(oldp, encoding="utf-8") as f:
cur = json.load(f)
targets = cur["frozen_targets"]["targets"]
old_hashes = [x["target_hash"] for x in new["frozen_targets"]["targets"]]
new_hashes = [x["target_hash"] for x in targets]
print("target_hash_list_byte_identical=", old_hashes == new_hashes)
print("declared_actual_classes=", cur["frozen_targets"]["count"], len(targets),
dict(sorted(Counter(x["class"] for x in targets).items())))
print("unique_target_hashes=", len(set(new_hashes)))
print("target_hash_format_valid=", all(len(x) == 64 and set(x) <= set("0123456789abcdef") for x in new_hashes))
pairsets=defaultdict(set)
for x in targets:
pairsets[x["class"]].add((x["a_sha"],x["b_sha"]))
for cls in ("natural","rename","copy"):
no_target=cur["sample"]["no_target"][cls]
all_pairs=pairsets[cls] | {(x["a_sha"],x["b_sha"]) for x in no_target}
print(f"{cls}: selected={cur['sample']['selected_pairs'][cls]} targetable={len(pairsets[cls])} no_target={len(no_target)} accounted={len(all_pairs)}")
print("hard_total=",cur["hard_challenge"]["frozen_hard_sample"]["total"])
seed=cur["seeds"]["truth_seed"]
side_bad=[]
for i, x in enumerate(targets):
if x["class"] == "natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",x["a_sha"],x["b_sha"]],
ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0] & 1 == 0 else "B"
else:
expected="A"
if x["side"] != expected:
side_bad.append((i,x["class"],x["side"],expected))
print("side_rule_mismatches=",side_bad)
pairs=set().union(*pairsets.values())
pairs |= {(x["a_sha"],x["b_sha"]) for c in ("natural","rename","copy") for x in cur["sample"]["no_target"][c]}
bad=[]
for a,b in pairs:
subprocess.run(["git","cat-file","-e",a+"^{commit}"],check=True)
subprocess.run(["git","cat-file","-e",b+"^{commit}"],check=True)
first=subprocess.check_output(["git","show","-s","--format=%P",b],text=True).split()[0]
if first != a: bad.append((a,b,first))
print("nonadjacent_or_wrong_direction=",bad)
PY
echo '== Frozen Step-1 provenance =='
git cat-file -e "${merge}^{commit}"
git cat-file -e "${reviewed}^{commit}"
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" \
"$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
done
echo '== Manifest structure =='
ast-grep outline "$manifest" --lang json --items all
echo '== Referenced contracts =='
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
echo "-- $path"
sed -n '1,260p' "$path"
doneLength of output: 32179 🏁 Script executed: #!/bin/bash
set -u -o pipefail
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Sidecar byte and verifier checks =='
python3 - <<'PY'
from pathlib import Path
p=Path("research/rh-m0/rh-m0-manifest.sha256")
data=p.read_bytes()
print("sidecar_bytes_repr=",repr(data))
print("contains_crlf=",b"\r\n" in data)
print("contains_bare_cr=",b"\r" in data.replace(b"\r\n",b""))
PY
echo '-- verifier from manifest directory --'
( cd research/rh-m0 && sha256sum --check rh-m0-manifest.sha256 ) || true
echo '-- verifier after only newline normalization, in manifest directory --'
tmp="$(mktemp)"
tr -d '\r' < "$sidecar" > "$tmp"
( cd research/rh-m0 && sha256sum --check "$tmp" ) || true
rm -f "$tmp"
echo '== JSON invariants, selection accounting, and forced-side rule =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
p="research/rh-m0/rh-m0-manifest.json"
old="cde73560f029235959b7775a8466fca059c3f34e:"+p
before=json.loads(subprocess.check_output(["git","show",old],text=True))
with open(p,encoding="utf-8") as f: m=json.load(f)
targets=m["frozen_targets"]["targets"]
print("json_valid=true")
print("target_hash_list_byte_identical=", [x["target_hash"] for x in before["frozen_targets"]["targets"]] == [x["target_hash"] for x in targets])
print("declared_actual_by_class=",m["frozen_targets"]["count"],len(targets),dict(sorted(Counter(x["class"] for x in targets).items())))
print("unique_target_hashes=",len({x["target_hash"] for x in targets}))
print("hash_formats_valid=",all(len(x["target_hash"])==64 and set(x["target_hash"])<=set("0123456789abcdef") for x in targets))
target_pairs=defaultdict(set)
for x in targets: target_pairs[x["class"]].add((x["a_sha"],x["b_sha"]))
all_pairs={}
for cls in ("natural","rename","copy"):
no=m["sample"]["no_target"][cls]
all_pairs[cls]=target_pairs[cls]|{(x["a_sha"],x["b_sha"]) for x in no}
print(f"{cls}_accounting=",m["sample"]["selected_pairs"][cls],len(target_pairs[cls]),len(no),len(all_pairs[cls]))
print("hard_total_declared=",m["hard_challenge"]["frozen_hard_sample"]["total"])
print("hard_total_actual=",sum(1 for x in targets if x["class"] in ("rename","copy")))
print("pair_class_overlaps=", {f"{a}:{b}":len(all_pairs[a]&all_pairs[b]) for a,b in (("natural","rename"),("natural","copy"),("rename","copy"))})
bad_side=[]
seed=m["seeds"]["truth_seed"]
for i,x in enumerate(targets):
if x["class"]=="natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",x["a_sha"],x["b_sha"]],ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0]&1==0 else "B"
else: expected="A"
if x["side"]!=expected: bad_side.append((i,x["class"],x["side"],expected))
print("side_mismatches=",bad_side)
bad_adj=[]
for a,b in set().union(*all_pairs.values()):
for x in (a,b): subprocess.run(["git","cat-file","-e",x+"^{commit}"],check=True)
parents=subprocess.check_output(["git","show","-s","--format=%P",b],text=True).split()
if not parents or parents[0]!=a: bad_adj.append((a,b,parents))
print("bad_first_parent_adjacency=",bad_adj)
print("all_composite_examples_five_axes=", all(set(x)==set(m["verdict_taxonomy"]["axes_are_five"]) for x in m["verdict_taxonomy"]["examples"]))
PY
echo '== Step-1 provenance =='
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
if git merge-base --is-ancestor "$reviewed" "$merge"; then echo 'reviewed_head_is_ancestor=true'; else echo 'reviewed_head_is_ancestor=false'; fi
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" "$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" "$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
done
echo '== Policy and contract text =='
python3 - <<'PY'
import json
paths=[
"research/rh-m0/rh-m0-manifest.json",
"contracts/finding-lineage-decision-v1.json",
"contracts/finding-lineage-v1.json",
]
for p in paths:
print("\n--",p)
with open(p,encoding="utf-8") as f: o=json.load(f)
if p.endswith("rh-m0-manifest.json"):
for k in ("baselines","branch_capability","hard_challenge","primary_holdout","rename_capability","truth_protocol","verdict_taxonomy"):
print(k+"=",json.dumps(o[k],indent=2,sort_keys=True))
else:
print(json.dumps(o,indent=2,sort_keys=True))
PYLength of output: 156934
The remaining completed checks passed:
I could not independently verify every external STS commit object and first-parent relationship. The sandbox does not contain STS object You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c0a8618f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ], | ||
| "status": "NO_TARGET entries remain in the frame as evidence of sample sparsity. They are NOT truth groups, they are never replaced, and no pair is substituted for them." | ||
| }, | ||
| "no_target_replacement_allowed": false, |
There was a problem hiding this comment.
When an experiment executor reads this dedicated boolean, false indicates that NO_TARGET replacement is allowed, contradicting the adjacent status text (they are never replaced) and the frozen pair-selection flags. Trusting this field would permit any of the 37 empty rename/copy pairs to be substituted after preregistration, changing the sampling frame and invalidating subsequent results; this should be true (or be renamed to a positive no_target_replacement_may_be_performed flag while remaining false).
Useful? React with 👍 / 👎.
…ssion External review flagged sample.no_target_replacement_allowed=false as ambiguous for a machine reader: a dedicated permission flag carrying false forces the reader to decide what a denied permission means. The same shape occurred twice more, once in the same object. Fixing only the flagged field would have left two identical ambiguities, so all three are converted: no_target_replacement_allowed = false -> no_target_replacement_prohibited = true pair_selection_may_be_modified = false -> pair_selection_modification_prohibited = true merge_detector_may_be_added = false -> merge_detector_addition_prohibited = true sample.boolean_naming_rule now records the convention, and the NO_TARGET status prose points at the new field instead of restating the rule in words. The 63 frozen target hashes are byte-identical. No target, sampling, truth or B3 surface is touched. The old field names survive only inside manifest_revision.findings_applied, where they describe the change. Manifest SHA-256 b113a31813636c31c2a78685631a8c4304f80bb13dd341151a604d0bc05d8d31 supersedes df0e9f8d7cac50b969ac5d53003885d2ad3d8c05584d011b9e0bcef09ef2c043. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
One review finding addressed, plus two of the same shape found alongside itNew exact HEAD The 63 frozen target hashes are byte-identical across all three revisions of this PR. Verified programmatically; the patch asserts it and would abort otherwise. P1, NO_TARGET replacement. Valid. A dedicated permission flag carrying The same shape occurred twice more, once in the very same object. Fixing only the flagged field would have left two identical ambiguities for the next round to find, so all three were converted:
Expected grep hit. The three old field names still appear in the raw file, only inside PR body corrected. It still advertised the revision-1 manifest hash while the file had moved on. Asking reviewers to check provenance while showing them stale provenance in the header is not a great look. It now reads Manifest SHA-256 No target, sampling, truth or B3 surface was touched. No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6555e68944
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "validated_requires": [ | ||
| ">= 3 adjudicable rename truth groups", | ||
| "zero unsafe relation errors on the adjudicable rename groups" |
There was a problem hiding this comment.
When at least three rename groups are adjudicable but B3 returns unresolved for every one, both listed requirements are satisfied because unresolved results are not unsafe relation errors. The pooled hard axis can separately gain its three correct resolutions entirely from copy targets, so the composite can report PASS-RENAME (and even HARD-PASS) without B3 resolving a single rename. Require a minimum number of correctly resolved rename groups before permitting PASS-RENAME.
Useful? React with 👍 / 👎.
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='6555e68944296c0f47dd5dbf52c59c76548f8422'
rev2='9c0a8618f131e1bf0362d80538bb2af41f2d2c9a'
rev1='cde73560f029235959b7775a8466fca059c3f34e'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Commit identity and scope =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$rev2"
git show -s --format='commit=%H%nparents=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=30 "$rev2" "$head" -- "$manifest" "$sidecar"
echo '== Sidecar portability and checksum =='
python3 - <<'PY'
from pathlib import Path
p=Path("research/rh-m0/rh-m0-manifest.sha256")
b=p.read_bytes()
print("sidecar_bytes=",repr(b))
print("has_cr=",b"\r" in b)
print("has_final_lf=",b.endswith(b"\n"))
PY
( cd research/rh-m0 && sha256sum --check rh-m0-manifest.sha256 )
sha256sum "$manifest"
cat "$sidecar"
echo '== Frozen target preservation and manifest invariants =='
python3 - <<'PY'
import hashlib,json,subprocess
from collections import Counter,defaultdict
p="research/rh-m0/rh-m0-manifest.json"
revs=[
"cde73560f029235959b7775a8466fca059c3f34e",
"9c0a8618f131e1bf0362d80538bb2af41f2d2c9a",
"6555e68944296c0f47dd5dbf52c59c76548f8422",
]
docs=[json.loads(subprocess.check_output(["git","show",f"{r}:{p}"],text=True)) for r in revs]
lists=[[x["target_hash"] for x in d["frozen_targets"]["targets"]] for d in docs]
print("target_lists_all_byte_identical=",lists[0]==lists[1]==lists[2])
m=docs[-1]
ts=m["frozen_targets"]["targets"]
print("revision=",m.get("manifest_revision",{}).get("revision"))
print("declared_actual_class_counts=",m["frozen_targets"]["count"],len(ts),dict(sorted(Counter(x["class"] for x in ts).items())))
print("unique_valid_target_hashes=",len(set(lists[-1])),all(len(x)==64 and set(x)<=set("0123456789abcdef") for x in lists[-1]))
pairs=defaultdict(set)
for x in ts: pairs[x["class"]].add((x["a_sha"],x["b_sha"]))
allpairs={}
for c in ("natural","rename","copy"):
no=m["sample"]["no_target"][c]
allpairs[c]=pairs[c]|{(x["a_sha"],x["b_sha"]) for x in no}
print(f"{c}_selection_accounting=",m["sample"]["selected_pairs"][c],len(pairs[c]),len(no),len(allpairs[c]))
print("pair_class_overlaps=",{
f"{a}:{b}":len(allpairs[a]&allpairs[b])
for a,b in (("natural","rename"),("natural","copy"),("rename","copy"))
})
print("hard_total_declared_actual=",m["hard_challenge"]["frozen_hard_sample"]["total"],sum(x["class"] in ("rename","copy") for x in ts))
seed=m["seeds"]["truth_seed"]
bad=[]
for i,x in enumerate(ts):
if x["class"]=="natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",x["a_sha"],x["b_sha"]],ensure_ascii=False,separators=(",",":")).encode()
want="A" if hashlib.sha256(raw).digest()[0]&1==0 else "B"
else: want="A"
if x["side"] != want: bad.append((i,x["class"],x["side"],want))
print("side_determination_mismatches=",bad)
old_names=("no_target_replacement_allowed","pair_selection_may_be_modified","merge_detector_may_be_added")
for name in old_names:
locations=[]
def walk(v,path="$"):
if isinstance(v,dict):
for k,w in v.items():
if k==name: locations.append(path+"."+k)
walk(w,path+"."+k)
elif isinstance(v,list):
for i,w in enumerate(v): walk(w,f"{path}[{i}]")
walk(m)
print(f"{name}_locations=",locations)
examples=m["verdict_taxonomy"]["examples"]
axes=set(m["verdict_taxonomy"]["axes_are_five"])
print("examples_have_exactly_five_axes=",all(set(e)==axes for e in examples))
PY
echo '== Step-1 provenance =='
git cat-file -e "${merge}^{commit}"
git cat-file -e "${reviewed}^{commit}"
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" \
"$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
done
echo '== Relevant manifest sections =='
python3 - <<'PY'
import json
p="research/rh-m0/rh-m0-manifest.json"
with open(p,encoding="utf-8") as f: m=json.load(f)
for key in ("manifest_revision","sample","seeds","baselines","truth_protocol","verdict_taxonomy","branch_capability","merge_capability","hard_challenge","primary_holdout"):
print("\n== "+key+" ==")
print(json.dumps(m[key],indent=2,sort_keys=True))
PYLength of output: 17790 🏁 Script executed: #!/bin/bash
set -euo pipefail
head='6555e68944296c0f47dd5dbf52c59c76548f8422'
rev2='9c0a8618f131e1bf0362d80538bb2af41f2d2c9a'
rev1='cde73560f029235959b7775a8466fca059c3f34e'
manifest='research/rh-m0/rh-m0-manifest.json'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Digest value after parsing sidecar fields =='
python3 - <<'PY'
from pathlib import Path
import hashlib
m=Path("research/rh-m0/rh-m0-manifest.json").read_bytes()
s=Path("research/rh-m0/rh-m0-manifest.sha256").read_bytes()
declared=s.split()[0].decode("ascii")
actual=hashlib.sha256(m).hexdigest()
print("declared=",declared)
print("actual=",actual)
print("digest_matches=",declared==actual)
PY
echo '== Targets, selection, side rule, and prohibitions =='
python3 - <<'PY'
import hashlib,json,subprocess
from collections import Counter, defaultdict
p="research/rh-m0/rh-m0-manifest.json"
commits=["cde73560f029235959b7775a8466fca059c3f34e","9c0a8618f131e1bf0362d80538bb2af41f2d2c9a","6555e68944296c0f47dd5dbf52c59c76548f8422"]
docs=[json.loads(subprocess.check_output(["git","show",f"{c}:{p}"],text=True)) for c in commits]
m=docs[-1]; ts=m["frozen_targets"]["targets"]
hs=[[t["target_hash"] for t in d["frozen_targets"]["targets"]] for d in docs]
print("targets_byte_identical_across_3_revisions=",hs[0]==hs[1]==hs[2])
print("counts=",m["frozen_targets"]["count"],len(ts),dict(Counter(t["class"] for t in ts)))
print("target_hashes_unique_and_valid=",len(set(hs[-1]),)==len(hs[-1]),all(len(h)==64 and set(h)<=set("0123456789abcdef") for h in hs[-1]))
byclass=defaultdict(set)
for t in ts: byclass[t["class"]].add((t["a_sha"],t["b_sha"]))
allpairs={}
for c in ("natural","rename","copy"):
no={(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][c]}
allpairs[c]=byclass[c]|no
print(f"{c}_selected_targetable_no_target_accounted=",m["sample"]["selected_pairs"][c],len(byclass[c]),len(no),len(allpairs[c]))
print("hard_declared_actual=",m["hard_challenge"]["frozen_hard_sample"]["total"],sum(t["class"]!="natural" for t in ts))
print("pair_class_overlap=", {a+":"+b:len(allpairs[a]&allpairs[b]) for a,b in (("natural","rename"),("natural","copy"),("rename","copy"))})
seed=m["seeds"]["truth_seed"]; bad=[]
for i,t in enumerate(ts):
if t["class"]=="natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",t["a_sha"],t["b_sha"]],ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0]&1==0 else "B"
else: expected="A"
if t["side"]!=expected: bad.append((i,t["class"],t["side"],expected))
print("side_mismatches=",bad)
print("required_prohibitions=",{
k:m["sample"].get(k) for k in ("no_target_replacement_prohibited","pair_selection_modification_prohibited")
}|{"merge_detector_addition_prohibited":m["merge_capability"].get("merge_detector_addition_prohibited")})
old=("no_target_replacement_allowed","pair_selection_may_be_modified","merge_detector_may_be_added")
def paths(v, name, at="$"):
r=[]
if isinstance(v,dict):
for k,w in v.items():
if k==name:r.append(at+"."+k)
r+=paths(w,name,at+"."+k)
elif isinstance(v,list):
for i,w in enumerate(v):r+=paths(w,name,f"{at}[{i}]")
return r
print("old_name_history_only=",{x:paths(m,x) for x in old})
print("truth_labels_created=",m["manifest_revision"]["truth_labels_created"])
print("b3_executed=",m["manifest_revision"]["b3_executed"])
axes=set(m["verdict_taxonomy"]["axes_are_five"])
print("every_composite_example_has_exactly_all_axes=",all(set(x)==axes for x in m["verdict_taxonomy"]["examples"]))
PY
echo '== Commit-object and first-parent checks for all listed pairs =='
python3 - <<'PY'
import json,subprocess
from collections import defaultdict
with open("research/rh-m0/rh-m0-manifest.json",encoding="utf-8") as f:m=json.load(f)
pairs=set()
for x in m["frozen_targets"]["targets"]:pairs.add((x["a_sha"],x["b_sha"]))
for c in ("natural","rename","copy"):
pairs|={(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][c]}
missing=[]; bad=[]
for a,b in sorted(pairs):
for sha in (a,b):
if subprocess.run(["git","cat-file","-e",sha+"^{commit}"]).returncode:missing.append(sha)
if a not in missing and b not in missing:
ps=subprocess.check_output(["git","show","-s","--format=%P",b],text=True).split()
if not ps or ps[0]!=a:bad.append((a,b,ps))
print("pair_count=",len(pairs))
print("missing_commit_objects=",sorted(set(missing)))
print("bad_first_parent_adjacency=",bad)
PY
echo '== Frozen Step-1 provenance =='
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" "$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" "$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
doneLength of output: 5517
The requested HEAD and parent match. The commit changes only the manifest and checksum sidecar. The follow-up validation timed out before it produced output. Therefore, I could not complete independent verification of target preservation, sample accounting, side determination, and Step-1 provenance for this revision. You are interacting with an AI system. |
External review found that PASS-RENAME required only >= 3 adjudicable rename groups plus zero unsafe relation errors. Four `unresolved` answers satisfy both, so the axis was reachable without resolving a single rename. The finding is valid and material. The identical vacuity existed on the branch axis, where "all endpoint sets exact" is trivially true when B3 produced no endpoint sets at all. It was not flagged, but it is the same defect class, so both axes are fixed together. Both are now exhaustive, precedence-ordered outcome tables: rename: 4 rows. The denominator counts only groups whose TRUTH is continued 1:1 across the rename, because correctly calling `ended` on a renamed file does not demonstrate R-CONT-RENAME. PASS requires >= 3 correct exact-counterpart resolutions. branch: 5 rows. PASS requires >= 3 correct resolutions with exact successor endpoint sets, spanning >= 2 revision pairs. verdict_taxonomy.no_vacuous_pass states the invariant once and records the resolution floor of every axis. natural and hard already carried floors, >= 20 B3-resolved and +3 over the better baseline; they needed no change. Also recorded: HARD-PASS pools rename and copy, so its +3 can be satisfied entirely by copy while no rename resolves. That is why rename has its own axis, and why HARD-PASS with RENAME-INCONCLUSIVE is a coherent composite rather than a contradiction. Both tables were verified by mechanical enumeration of every reachable state: zero uncovered states, zero vacuous PASS. Manifest SHA-256 2b8ce7018b6a67a6a522ac8d77638b4ccbd3f76634aa15b2f790443d0e6ad614 supersedes b113a31813636c31c2a78685631a8c4304f80bb13dd341151a604d0bc05d8d31. The 63 frozen target hashes are byte-identical. No sampling, truth or B3 surface was touched. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
The rename axis was passable without resolving a single renameNew exact HEAD The finding is valid and material. The identical vacuity existed on the branch axis and was not flagged: row "all endpoint sets exact" is trivially true when B3 produced no endpoint sets at all. Same defect class, so both are fixed together rather than leaving the next round an identical finding. Both axes are now exhaustive, precedence-ordered outcome tables. rename, 4 rows. The denominator counts only groups whose truth is branch, 5 rows.
Also recorded explicitly: Verified by mechanical enumeration of every reachable state of both tables: zero uncovered states, zero vacuous PASS. The exact case raised in review — 4 adjudicable, 0 resolved, 0 unsafe — now yields Manifest SHA-256 The 63 frozen target hashes are byte-identical across all four revisions. No sampling, truth or B3 surface was touched. No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='03d56ace0623db64abe9db9d88c385717968ccca'
parent='6555e68944296c0f47dd5dbf52c59c76548f8422'
rev1='cde73560f029235959b7775a8466fca059c3f34e'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Commit identity and scope =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparents=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=25 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Checksum sidecar =='
python3 - <<'PY'
from pathlib import Path
import hashlib
manifest=Path("research/rh-m0/rh-m0-manifest.json").read_bytes()
sidecar=Path("research/rh-m0/rh-m0-manifest.sha256").read_bytes()
print("sidecar_bytes=",repr(sidecar))
print("has_carriage_return=",b"\r" in sidecar)
print("has_final_lf=",sidecar.endswith(b"\n"))
declared=sidecar.split()[0].decode("ascii")
actual=hashlib.sha256(manifest).hexdigest()
print("declared_digest=",declared)
print("actual_digest=",actual)
print("digest_matches=",declared==actual)
PY
( cd research/rh-m0 && sha256sum --check rh-m0-manifest.sha256 )
echo '== Manifest outline =='
ast-grep outline "$manifest" --lang json --items all
echo '== Target preservation, sampling, sides, and outcome-table structure =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
p="research/rh-m0/rh-m0-manifest.json"
commits=[
"cde73560f029235959b7775a8466fca059c3f34e",
"9c0a8618f131e1bf0362d80538bb2af41f2d2c9a",
"6555e68944296c0f47dd5dbf52c59c76548f8422",
"03d56ace0623db64abe9db9d88c385717968ccca",
]
docs=[json.loads(subprocess.check_output(["git","show",f"{c}:{p}"],text=True)) for c in commits]
m=docs[-1]
targets=m["frozen_targets"]["targets"]
hashes=[[x["target_hash"] for x in d["frozen_targets"]["targets"]] for d in docs]
print("target_hash_lists_byte_identical_across_four_revisions=",hashes[0]==hashes[1]==hashes[2]==hashes[3])
print("manifest_revision=",m.get("manifest_revision",{}).get("revision"))
print("target_count_declared_actual=",m["frozen_targets"]["count"],len(targets))
print("target_class_counts=",dict(sorted(Counter(x["class"] for x in targets).items())))
print("unique_valid_target_hashes=",len(set(hashes[-1]))==len(hashes[-1]),
all(len(h)==64 and set(h)<=set("0123456789abcdef") for h in hashes[-1]))
byclass=defaultdict(set)
for t in targets: byclass[t["class"]].add((t["a_sha"],t["b_sha"]))
allpairs={}
for c in ("natural","rename","copy"):
no={(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][c]}
allpairs[c]=byclass[c]|no
print(f"{c}_selected_targetable_no_target_accounted=",
m["sample"]["selected_pairs"][c],len(byclass[c]),len(no),len(allpairs[c]))
print("hard_total_declared_actual=",
m["hard_challenge"]["frozen_hard_sample"]["total"],
sum(t["class"] in ("rename","copy") for t in targets))
print("pair_class_overlaps=",{
a+":"+b:len(allpairs[a]&allpairs[b])
for a,b in (("natural","rename"),("natural","copy"),("rename","copy"))
})
seed=m["seeds"]["truth_seed"]
mismatches=[]
for i,t in enumerate(targets):
if t["class"]=="natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",t["a_sha"],t["b_sha"]],
ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0]&1==0 else "B"
else:
expected="A"
if t["side"]!=expected:
mismatches.append((i,t["class"],t["side"],expected))
print("side_determination_mismatches=",mismatches)
print("truth_labels_created=",m["manifest_revision"].get("truth_labels_created"))
print("b3_executed=",m["manifest_revision"].get("b3_executed"))
for key in ("rename_capability","branch_capability","verdict_taxonomy"):
print("\n== "+key+" ==")
print(json.dumps(m.get(key),indent=2,sort_keys=True))
PY
echo '== Frozen Step-1 provenance available locally =='
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
if git merge-base --is-ancestor "$reviewed" "$merge"; then
echo 'reviewed_head_is_ancestor=true'
else
echo 'reviewed_head_is_ancestor=false'
fi
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" \
"$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
done
echo '== Listed commit objects and first-parent adjacency =='
python3 - <<'PY'
import json, subprocess
with open("research/rh-m0/rh-m0-manifest.json",encoding="utf-8") as f: m=json.load(f)
pairs={(x["a_sha"],x["b_sha"]) for x in m["frozen_targets"]["targets"]}
for c in ("natural","rename","copy"):
pairs|={(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][c]}
missing=[]; bad=[]
for a,b in sorted(pairs):
available=True
for sha in (a,b):
if subprocess.run(["git","cat-file","-e",sha+"^{commit}"]).returncode:
missing.append(sha); available=False
if available:
parents=subprocess.check_output(["git","show","-s","--format=%P",b],text=True).split()
if not parents or parents[0]!=a: bad.append((a,b,parents))
print("pair_count=",len(pairs))
print("missing_commit_objects=",sorted(set(missing)))
print("bad_first_parent_adjacency=",bad)
PYLength of output: 25438
Convert the sidecar to LF line endings. Then run The exact HEAD and parent match the requested commits. The commit changes only the manifest and checksum sidecar. The script stopped at the checksum verification failure. Therefore, target preservation, verdict-table validation, and provenance checks remain inconclusive for this revision. You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03d56ace06
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| "condition": "any unsafe relation error on an adjudicable rename target", | ||
| "row": 1, | ||
| "verdict": "FAIL/PIVOT" |
There was a problem hiding this comment.
When any adjudicable rename target has an unsafe relation error, this row emits FAIL/PIVOT, but verdict_taxonomy.axes.rename permits only PASS-RENAME, RENAME-INCONCLUSIVE, and FAIL. The resulting five-axis composite therefore cannot both follow this outcome table and satisfy the frozen taxonomy; use the taxonomy's FAIL value here or add FAIL/PIVOT to the allowed rename verdicts before execution.
Useful? React with 👍 / 👎.
rename_capability.outcome_table row 1 emitted FAIL/PIVOT while verdict_taxonomy.axes.rename declares PASS-RENAME / RENAME-INCONCLUSIVE / FAIL. No run could satisfy both. The mirror image of the same defect was present and unreported: FAIL was declared for the rename axis but reachable from no row. The row now emits FAIL. The rename token set is deliberately NOT widened to FAIL/PIVOT. That compound belongs to the branch axis by preregistration, and a result vocabulary that grows to fit one row stops discriminating. The pivot semantics are preserved as an experiment disposition instead: axis verdict FAIL -> disposition PIVOT axis verdict INCONCLUSIVE -> disposition HOLD axis verdict PASS -> disposition PROCEED A verdict records what was measured; a disposition records what the project does next. Fusing them makes the measured result depend on a management decision. The branch axis keeps its compound FAIL/PIVOT token. It was preregistered that way and is internally consistent with its own token set, so it is recorded as a known wart rather than silently rewritten. Editing a frozen token to satisfy a style preference is the kind of post-hoc change this manifest exists to prevent. verdict_taxonomy.consistency_rule now states the invariant: every verdict in any outcome_table must belong to its axis token set, and every declared token should be reachable. Verified mechanically in both directions, zero mismatches and zero unreachable tokens. Enumerating every reachable rename state yields exactly the three declared verdicts. Why prose review missed it: the rename table was written by analogy with the branch table, which legitimately uses FAIL/PIVOT. Reading compares a row against its neighbours; only a mechanical check compares it against its own axis. Manifest SHA-256 f4c5181eaf0948dc38f6d86c884404331a5111f275228d78cd7ca22c0f97f47e supersedes 2b8ce7018b6a67a6a522ac8d77638b4ccbd3f76634aa15b2f790443d0e6ad614. The 63 frozen target hashes are byte-identical. No sampling, truth or B3 surface was touched. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
One verdict token, plus the check that would have caught itNew exact HEAD Valid. Running the check mechanically surfaced the mirror image of the same defect, which had not been reported: Fixed minimally. Row 1 now emits Pivot semantics kept, but separated from the measurement:
A verdict records what was measured; a disposition records what the project does next. Fusing them makes the measured result depend on a management decision, and groups badly for any consumer. The branch axis keeps its compound token. It was preregistered that way and is internally consistent with its own token set, so it is recorded as a known wart rather than silently rewritten. Editing a frozen token to satisfy a style preference is precisely the post-hoc change this manifest exists to prevent. The invariant is now written down. Why prose review missed it: the rename table was written by analogy with the branch table, which legitimately uses Manifest SHA-256 The 63 frozen target hashes are byte-identical across all five revisions. No sampling, truth or B3 surface was touched. No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa'
parent='03d56ace0623db64abe9db9d88c385717968ccca'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
revisions=(
'cde73560f029235959b7775a8466fca059c3f34e'
'9c0a8618f131e1bf0362d80538bb2af41f2d2c9a'
'6555e68944296c0f47dd5dbf52c59c76548f8422'
'03d56ace0623db64abe9db9d88c385717968ccca'
'5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa'
)
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Commit identity and change scope =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=12 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Sidecar bytes and standard verification =='
python3 - <<'PY'
from pathlib import Path
import hashlib
manifest = Path("research/rh-m0/rh-m0-manifest.json").read_bytes()
sidecar = Path("research/rh-m0/rh-m0-manifest.sha256").read_bytes()
print("sidecar_repr=", repr(sidecar))
print("contains_carriage_return=", b"\r" in sidecar)
print("ends_with_lf=", sidecar.endswith(b"\n"))
fields = sidecar.split()
print("sidecar_field_count=", len(fields))
print("declared_digest=", fields[0].decode("ascii") if fields else None)
print("actual_digest=", hashlib.sha256(manifest).hexdigest())
print("digest_matches=", bool(fields) and fields[0].decode("ascii") == hashlib.sha256(manifest).hexdigest())
PY
( cd research/rh-m0 && sha256sum --check rh-m0-manifest.sha256 )
echo '== Frozen-target preservation and structural accounting =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
p = "research/rh-m0/rh-m0-manifest.json"
revs = [
"cde73560f029235959b7775a8466fca059c3f34e",
"9c0a8618f131e1bf0362d80538bb2af41f2d2c9a",
"6555e68944296c0f47dd5dbf52c59c76548f8422",
"03d56ace0623db64abe9db9d88c385717968ccca",
"5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa",
]
docs = [json.loads(subprocess.check_output(["git","show",f"{r}:{p}"], text=True)) for r in revs]
m = docs[-1]
targets = m["frozen_targets"]["targets"]
hash_lists = [[x["target_hash"] for x in d["frozen_targets"]["targets"]] for d in docs]
print("target_hash_lists_byte_identical_all_revisions=", all(v == hash_lists[0] for v in hash_lists[1:]))
print("manifest_revision=", m["manifest_revision"]["revision"])
print("declared_actual_target_count=", m["frozen_targets"]["count"], len(targets))
print("target_class_counts=", dict(sorted(Counter(x["class"] for x in targets).items())))
print("unique_target_hashes=", len(set(hash_lists[-1])))
print("target_hashes_valid=", all(len(h)==64 and set(h) <= set("0123456789abcdef") for h in hash_lists[-1]))
by_class=defaultdict(set)
for t in targets: by_class[t["class"]].add((t["a_sha"],t["b_sha"]))
all_pairs={}
for cls in ("natural","rename","copy"):
no_target={(x["a_sha"],x["b_sha"]) for x in m["sample"]["no_target"][cls]}
all_pairs[cls]=by_class[cls] | no_target
print(f"{cls}_selected_targetable_no_target_accounted=", m["sample"]["selected_pairs"][cls],len(by_class[cls]),len(no_target),len(all_pairs[cls]))
print("hard_declared_actual=",m["hard_challenge"]["frozen_hard_sample"]["total"],sum(t["class"] != "natural" for t in targets))
print("cross_class_pair_overlaps=", {f"{a}:{b}":len(all_pairs[a]&all_pairs[b]) for a,b in (("natural","rename"),("natural","copy"),("rename","copy"))})
seed=m["seeds"]["truth_seed"]
bad_sides=[]
for i,t in enumerate(targets):
if t["class"] == "natural":
raw=json.dumps(["rh-m0-side/v1",seed,"sts",t["a_sha"],t["b_sha"]],ensure_ascii=False,separators=(",",":")).encode()
expected="A" if hashlib.sha256(raw).digest()[0]&1 == 0 else "B"
else:
expected="A"
if t["side"] != expected: bad_sides.append((i,t["class"],t["side"],expected))
print("side_determination_mismatches=",bad_sides)
print("truth_labels_created=",m["manifest_revision"]["truth_labels_created"])
print("b3_executed=",m["manifest_revision"]["b3_executed"])
PY
echo '== Verdict vocabulary consistency and relevant sections =='
python3 - <<'PY'
import json
p="research/rh-m0/rh-m0-manifest.json"
with open(p,encoding="utf-8") as f: m=json.load(f)
taxonomy=m["verdict_taxonomy"]
print("consistency_rule=",taxonomy.get("consistency_rule"))
for capability_name in ("rename_capability","branch_capability"):
capability=m[capability_name]
axis=capability["axis"]
declared=set(taxonomy["axes"][axis]["tokens"])
rows=capability["outcome_table"]
emitted={row["verdict"] for row in rows}
print(f"{capability_name}_axis=",axis)
print(f"{capability_name}_declared_tokens=",sorted(declared))
print(f"{capability_name}_emitted_tokens=",sorted(emitted))
print(f"{capability_name}_out_of_vocabulary=",sorted(emitted-declared))
print(f"{capability_name}_unreachable_declared=",sorted(declared-emitted))
print(json.dumps(capability,indent=2,sort_keys=True))
print("no_vacuous_pass=",taxonomy.get("no_vacuous_pass"))
print("axes_are_five=",taxonomy.get("axes_are_five"))
print("examples_exact_axes=",all(set(x)==set(taxonomy["axes_are_five"]) for x in taxonomy["examples"]))
PY
echo '== Frozen Step-1 provenance =='
git show -s --format='merge=%H%nparents=%P%ncommitter_time=%cI' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
done
for path in aggregate/normalize.py aggregate/provenance.py aggregate/sarif_read.py; do
printf '%s parent1=%s parent2=%s merge=%s\n' "$path" \
"$(git rev-parse "$merge^1:$path")" "$(git rev-parse "$merge^2:$path")" "$(git rev-parse "$merge:$path")"
doneLength of output: 18693
Convert this sidecar to LF line endings. Then run this command from The requested HEAD and parent match. The commit changes only the manifest and checksum sidecar. The verification script stopped at the checksum failure. Therefore, verification of target preservation across all five revisions, sample accounting, side determination, and verdict-table consistency is incomplete for this revision. You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c89ba515d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "method": "cross-product check of all outcome_table rows against verdict_taxonomy.axes, in both directions", | ||
| "result_at_this_revision": { | ||
| "declared_but_unreachable": 0, | ||
| "mismatches": 0 |
There was a problem hiding this comment.
The stated mechanical method cannot produce declared_but_unreachable: 0 across all five axes: only branch_capability and rename_capability contain an outcome_table, while the hard, natural, and merge axes collectively declare seven tokens that appear in no outcome-table row. At exact HEAD 5c89ba5, enumerating each axis's declared tokens against its corresponding outcome_table therefore reports those seven as unreachable. Either restrict the invariant/result explicitly to the two table-backed axes, or encode/check equivalent outcome rows for hard, natural, and merge before claiming a bidirectional zero.
Useful? React with 👍 / 👎.
Documentation consistency onlyNew exact HEAD Valid. Two sentences inside Current state, derived from the structure: four axes are table-backed, and applying table reachability to all five would falsely report exactly one unreachable token — The count is no longer prose. It moved into Two structural invariants are now asserted — against the structure, never against any wording: Asserting the wording would require a checker for the checker's description, and then one for that. Nothing else moved. Asserted before writing: Verified from the committed blobs: zero carriage returns in either file, Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='11057180fe0c12011ea82c7f85cb57483c0225da'
parent='ad58b29383d36b9fad0f0dc65a5faa3ace6636e8'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
echo '== Commit identity and changed paths =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=30 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Committed-blob checksum and line endings =='
python3 - <<'PY'
import hashlib, subprocess, tempfile
from pathlib import Path
commit = "11057180fe0c12011ea82c7f85cb57483c0225da"
manifest = "research/rh-m0/rh-m0-manifest.json"
sidecar = "research/rh-m0/rh-m0-manifest.sha256"
m = subprocess.check_output(["git", "show", f"{commit}:{manifest}"])
s = subprocess.check_output(["git", "show", f"{commit}:{sidecar}"])
print("manifest_has_cr=", b"\r" in m)
print("sidecar_has_cr=", b"\r" in s)
print("manifest_sha256=", hashlib.sha256(m).hexdigest())
print("sidecar=", repr(s))
with tempfile.TemporaryDirectory() as d:
root = Path(d)
(root / "rh-m0-manifest.json").write_bytes(m)
(root / "rh-m0-manifest.sha256").write_bytes(s)
r = subprocess.run(["sha256sum", "--check", "rh-m0-manifest.sha256"],
cwd=d, capture_output=True, text=True)
print("sha256sum_exit=", r.returncode)
print("sha256sum_stdout=", r.stdout.strip())
print("sha256sum_stderr=", r.stderr.strip())
PY
echo '== Revision-preservation and consistency-rule structure =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter
p="research/rh-m0/rh-m0-manifest.json"
before_commit="ad58b29383d36b9fad0f0dc65a5faa3ace6636e8"
after_commit="11057180fe0c12011ea82c7f85cb57483c0225da"
before_raw=subprocess.check_output(["git","show",f"{before_commit}:{p}"])
after_raw=subprocess.check_output(["git","show",f"{after_commit}:{p}"])
before=json.loads(before_raw)
after=json.loads(after_raw)
print("revision=",after["manifest_revision"]["revision"])
print("target_hash_lists_identical=", [x["target_hash"] for x in before["frozen_targets"]["targets"]] == [x["target_hash"] for x in after["frozen_targets"]["targets"]])
print("target_counts=",after["frozen_targets"]["count"],len(after["frozen_targets"]["targets"]),dict(Counter(t["class"] for t in after["frozen_targets"]["targets"])))
for key in ("axes","axis_disposition"):
print("taxonomy_"+key+"_unchanged=",before["verdict_taxonomy"][key] == after["verdict_taxonomy"][key])
for key in ("truth_protocol","seeds","primary_holdout","hard_challenge","rename_capability","branch_capability"):
print(key+"_unchanged=",before[key] == after[key])
tax=after["verdict_taxonomy"]
rule=tax["consistency_rule"]
axes=set(tax["axes"])
table_sections={
"natural":"primary_holdout",
"hard":"hard_challenge",
"rename":"rename_capability",
"branch":"branch_capability",
"merge":"merge_capability",
}
table_backed={axis for axis, section in table_sections.items() if "outcome_table" in after[section]}
scope=set(rule["scope"])
non_table=set(rule["non_table_backed_axes"])
print("table_backed_axes=",sorted(table_backed))
print("scope=",sorted(scope))
print("non_table_backed_axes=",sorted(non_table))
print("scope_equals_table_backed=",scope == table_backed)
print("non_table_equals_axes_minus_scope=",non_table == axes-scope)
emitted=set()
for axis in scope:
emitted |= {row["verdict"] for row in after[table_sections[axis]]["outcome_table"]}
falsely_unreachable=sorted(
token for axis in axes-scope for token in after["verdict_taxonomy"]["axes"][axis]
if token not in emitted
)
print("computed_falsely_unreachable=",falsely_unreachable)
print("derived_field=",rule.get("falsely_unreachable_if_applied_to_all_axes"))
print("derived_field_matches=",rule.get("falsely_unreachable_if_applied_to_all_axes") == falsely_unreachable)
for entry in after["manifest_revision"]["history"]:
raw=subprocess.check_output(["git","show",f"{entry['commit']}:{p}"])
print("history_sha",entry["revision"],hashlib.sha256(raw).hexdigest() == entry["sha256"])
print("truth_labels_created=",after["manifest_revision"]["truth_labels_created"])
print("b3_executed=",after["manifest_revision"]["b3_executed"])
PY
echo '== Relevant current text =='
python3 - <<'PY'
import json
with open("research/rh-m0/rh-m0-manifest.json", encoding="utf-8") as f:
m=json.load(f)
print(json.dumps(m["manifest_revision"], indent=2, sort_keys=True))
print(json.dumps(m["verdict_taxonomy"]["consistency_rule"], indent=2, sort_keys=True))
PYLength of output: 30566 🏁 Script executed: #!/bin/bash
set -euo pipefail
manifest='research/rh-m0/rh-m0-manifest.json'
echo '== Derived-field location =='
rg -n -C 12 '"falsely_unreachable_if_applied_to_all_axes"|"derived": true|"why_derived"' "$manifest"
echo '== References outside the manifest =='
rg -n -C 3 \
'falsely_unreachable_if_applied_to_all_axes|scope_equals_table_backed_axes|non_table_backed_is_the_complement|verified_mechanically' \
. -g '!research/rh-m0/rh-m0-manifest.json'
echo '== Value-level validation =='
python3 - <<'PY'
import json
with open("research/rh-m0/rh-m0-manifest.json", encoding="utf-8") as f:
m=json.load(f)
tax=m["verdict_taxonomy"]
rule=tax["consistency_rule"]
section={"natural":"primary_holdout","hard":"hard_challenge","rename":"rename_capability","branch":"branch_capability","merge":"merge_capability"}
axes=set(tax["axes"])
table_backed={axis for axis, name in section.items() if "outcome_table" in m[name]}
non_table=axes-table_backed
expected=[
f"{axis}: {token}"
for axis in sorted(non_table)
for token in tax["axes"][axis]
]
actual=rule["falsely_unreachable_if_applied_to_all_axes"]
print("expected_count=",len(expected))
print("actual_count=",actual["count"])
print("expected_tokens=",expected)
print("actual_tokens=",actual["tokens"])
print("content_matches=",actual["count"]==len(expected) and actual["tokens"]==expected)
PYLength of output: 5261 🏁 Script executed: #!/bin/bash
set -u -o pipefail
echo '== Files in the RH-M0 directory =='
fd -HI -t f . research/rh-m0 | sort
echo '== Nonfatal external-reference search =='
rg -n -C 3 \
'falsely_unreachable_if_applied_to_all_axes|scope_equals_table_backed_axes|non_table_backed_is_the_complement|verified_mechanically' \
. -g '!research/rh-m0/rh-m0-manifest.json' || true
echo '== Value-level validation =='
python3 - <<'PY'
import json
with open("research/rh-m0/rh-m0-manifest.json", encoding="utf-8") as f:
m=json.load(f)
tax=m["verdict_taxonomy"]
rule=tax["consistency_rule"]
section={"natural":"primary_holdout","hard":"hard_challenge","rename":"rename_capability","branch":"branch_capability","merge":"merge_capability"}
axes=set(tax["axes"])
table_backed={axis for axis, name in section.items() if "outcome_table" in m[name]}
non_table=axes-table_backed
expected=[
f"{axis}: {token}"
for axis in sorted(non_table)
for token in tax["axes"][axis]
]
actual=rule["falsely_unreachable_if_applied_to_all_axes"]
print("expected_count=",len(expected))
print("actual_count=",actual["count"])
print("expected_tokens=",expected)
print("actual_tokens=",actual["tokens"])
print("content_matches=",actual["count"]==len(expected) and actual["tokens"]==expected)
PYLength of output: 1727
The remaining checks passed:
You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11057180fe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "count": 1, | ||
| "derivation": "for every axis without an outcome_table, every token declared for it in verdict_taxonomy.axes", | ||
| "derived": true, |
There was a problem hiding this comment.
At exact HEAD 1105718, this is still a cached JSON literal rather than a derived value: repo-wide search finds no generator or validator for this field, and the new structural invariants are likewise unevaluated strings followed by verified_mechanically: true. Consequently, adding another outcome table or taxonomy token can make count and tokens stale exactly as before; derived: true alone does not make the claimed invariant hold. Add an executable check/generation step or avoid storing the redundant calculated values.
Useful? React with 👍 / 👎.
…ain it Both external reviewers independently found that falsely_unreachable_if_applied_to_all_axes stored a literal count and token list while declaring derived: true, with no generator committed anywhere in the repository. The arithmetic was correct and both reviewers confirmed the value. The claim about its provenance was not: a reader of this repository sees a JSON literal and a declaration that something derives it, and nothing does. The materialised current value is removed rather than backed by a new checker. current_value_materialization is now PROHIBITED, with the derivation rule kept so a consumer can compute the value from the authoritative structure. The revision-6 entry stays, because it records what the computation yielded then and cannot become semantically stale. why_the_scope_matters no longer names the number at all. Stating it in a third place would only schedule the next archaeological layer. The same seam ran wider than the finding. Four verified_mechanically flags and three result snapshots read as claims that a permanent validator enforces those properties. RH-M0 commits the manifest and its checksum sidecar only, so nothing does. They are renamed to say what they are: verified_mechanically -> verified_during_revision_construction mechanical_result -> result_at_revision_construction result_at_this_revision -> result_at_revision_construction verdict_taxonomy.verification_status states the position once: these are construction-time observations, not live repository invariants, and no checker is committed to service one counter. Documentation only. Asserted before writing: verdict_taxonomy.axes, axis_disposition, consistency_rule.scope, all four outcome tables, truth_protocol, seeds and the 63 frozen target hashes. A key-level scan confirms none of the retired names survive as keys; they appear only inside prose that describes the rename. Both files remain LF; sha256sum --check passes. Manifest SHA-256 08f266e927d733ac177a6b5592da6cab4db51df8b82467e0bacc3b7f6b342758 supersedes 502477c44fb9a315efef10d3504ef33eabfa12f17401ed4abc7905c5c9c50b94. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
The cache is removed, not managedNew exact HEAD Valid, and raised independently by both reviewers. Removed rather than backed by a new checker. Committing a validator to service one counter would have re-expanded the RH-M0 surface at the exact moment the remaining defects became purely meta-level. The seam ran wider than the finding. Four
Two things worth noting about how this was verified. The patch's first assertion attempt scanned the serialised JSON for the retired names and fired — because those names legitimately appear inside the prose describing the rename. The assertion was rewritten to scan keys, which is the property that actually matters, and the run aborted before writing rather than producing a half-correct artifact. A leftover Verified from the committed blobs: zero carriage returns in either file, Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='b10d48b231de80fa4a5deeb636174a555188a74b'
parent='11057180fe0c12011ea82c7f85cb57483c0225da'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Commit identity and changed paths =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=20 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Committed-blob checksum and encoding =='
python3 - <<'PY'
import hashlib, subprocess, tempfile
from pathlib import Path
commit='b10d48b231de80fa4a5deeb636174a555188a74b'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
m=subprocess.check_output(['git','show',f'{commit}:{manifest}'])
s=subprocess.check_output(['git','show',f'{commit}:{sidecar}'])
print('manifest_has_cr=', b'\r' in m)
print('sidecar_has_cr=', b'\r' in s)
print('manifest_sha256=', hashlib.sha256(m).hexdigest())
print('sidecar_repr=', repr(s))
with tempfile.TemporaryDirectory() as d:
root=Path(d)
(root/'rh-m0-manifest.json').write_bytes(m)
(root/'rh-m0-manifest.sha256').write_bytes(s)
r=subprocess.run(['sha256sum','--check','rh-m0-manifest.sha256'],
cwd=d,capture_output=True,text=True)
print('sha256sum_exit=',r.returncode)
print('sha256sum_stdout=',r.stdout.strip())
print('sha256sum_stderr=',r.stderr.strip())
PY
echo '== Revision-10 semantic, vocabulary, and target invariants =='
python3 - <<'PY'
import hashlib, json, subprocess
from collections import Counter, defaultdict
p='research/rh-m0/rh-m0-manifest.json'
revs=[
'cde73560f029235959b7775a8466fca059c3f34e',
'9c0a8618f131e1bf0362d80538bb2af41f2d2c9a',
'6555e68944296c0f47dd5dbf52c59c76548f8422',
'03d56ace0623db64abe9db9d88c385717968ccca',
'5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa',
'ac54fcb3420550f0423366842fdd488f69bf1b3a',
'6e3304a498951089f8945f6cb3ec99978e65e887',
'ad58b29383d36b9fad0f0dc65a5faa3ace6636e8',
'11057180fe0c12011ea82c7f85cb57483c0225da',
'b10d48b231de80fa4a5deeb636174a555188a74b',
]
docs=[json.loads(subprocess.check_output(['git','show',f'{r}:{p}'])) for r in revs]
before, m=docs[-2], docs[-1]
targets=m['frozen_targets']['targets']
lists=[[t['target_hash'] for t in d['frozen_targets']['targets']] for d in docs]
print('json_valid=true')
print('revision=',m['manifest_revision']['revision'])
print('target_hash_lists_identical_all_10=',all(x==lists[0] for x in lists[1:]))
print('target_counts=',m['frozen_targets']['count'],len(targets),
dict(sorted(Counter(t['class'] for t in targets).items())))
print('target_hashes_unique_valid=',len(set(lists[-1]))==len(lists[-1]) and all(
len(h)==64 and set(h)<=set('0123456789abcdef') for h in lists[-1]))
pairs=defaultdict(set)
for t in targets: pairs[t['class']].add((t['a_sha'],t['b_sha']))
for cls in ('natural','rename','copy'):
no={(x['a_sha'],x['b_sha']) for x in m['sample']['no_target'][cls]}
print(cls+'_selected_targetable_no_target_accounted=',
m['sample']['selected_pairs'][cls],len(pairs[cls]),len(no),len(pairs[cls]|no))
print('hard_declared_actual=',m['hard_challenge']['frozen_hard_sample']['total'],
sum(t['class']!='natural' for t in targets))
seed=m['seeds']['truth_seed']; bad_sides=[]
for i,t in enumerate(targets):
if t['class']=='natural':
raw=json.dumps(['rh-m0-side/v1',seed,'sts',t['a_sha'],t['b_sha']],
ensure_ascii=False,separators=(',',':')).encode()
expected='A' if hashlib.sha256(raw).digest()[0]&1==0 else 'B'
else: expected='A'
if t['side']!=expected: bad_sides.append((i,t['class'],t['side'],expected))
print('side_determination_mismatches=',bad_sides)
print('truth_labels_created=',m['manifest_revision']['truth_labels_created'])
print('b3_executed=',m['manifest_revision']['b3_executed'])
# Assert the documented unchanged surfaces.
for key in ('truth_protocol','seeds','primary_holdout','hard_challenge',
'rename_capability','branch_capability'):
print(key+'_unchanged_from_rev9=',before[key]==m[key])
for key in ('axes','axis_disposition'):
print('taxonomy_'+key+'_unchanged_from_rev9=',
before['verdict_taxonomy'][key]==m['verdict_taxonomy'][key])
# Retired keys must not survive as keys; prose may legitimately quote them.
retired={'falsely_unreachable_if_applied_to_all_axes','derived','why_derived',
'verified_mechanically','mechanical_result','result_at_this_revision'}
found=[]
def walk(x,path='$'):
if isinstance(x,dict):
for k,v in x.items():
if k in retired: found.append(path+'.'+k)
walk(v,path+'.'+k)
elif isinstance(x,list):
for i,v in enumerate(x): walk(v,f'{path}[{i}]')
walk(m)
print('retired_key_locations=',found)
tax=m['verdict_taxonomy']
status=tax.get('verification_status')
print('verification_status=',json.dumps(status,sort_keys=True))
rule=tax['consistency_rule']
print('current_value_materialization=',rule.get('current_value_materialization'))
print('consistency_rule_scope=',rule.get('scope'))
print('non_table_backed_axes=',rule.get('non_table_backed_axes'))
axes=tax['axes']
sections={'natural':'primary_holdout','hard':'hard_challenge',
'rename':'rename_capability','branch':'branch_capability',
'merge':'merge_capability'}
table_backed={a for a,s in sections.items() if 'outcome_table' in m[s]}
print('scope_equals_table_backed=',set(rule['scope'])==table_backed)
print('non_table_equals_complement=',set(rule['non_table_backed_axes'])==set(axes)-table_backed)
declared={t for vals in axes.values() for t in vals}
mapping=tax['axis_disposition']['mapping']
print('tokens_unmapped=',sorted(declared-set(mapping)))
print('mapped_but_undeclared=',sorted(set(mapping)-declared))
for axis in table_backed:
emitted={r['verdict'] for r in m[sections[axis]]['outcome_table']}
print(axis+'_emitted_not_declared=',sorted(emitted-set(axes[axis])))
print(axis+'_declared_unreachable=',sorted(set(axes[axis])-emitted))
vocab=set(m['truth_protocol']['label_vocabulary'])
for name in ('rename_capability','branch_capability'):
print(name+'_unknown_truth_refs=',sorted(set(m[name].get('truth_labels_referenced',[]))-vocab))
PY
echo '== Revision-history hashes and frozen Step-1 provenance =='
python3 - <<'PY'
import hashlib,json,subprocess
p='research/rh-m0/rh-m0-manifest.json'
m=json.loads(subprocess.check_output(['git','show','b10d48b231de80fa4a5deeb636174a555188a74b:'+p]))
for e in m['manifest_revision']['history']:
raw=subprocess.check_output(['git','show',f"{e['commit']}:{p}"])
print('history_sha',e['revision'],hashlib.sha256(raw).hexdigest()==e['sha256'])
PY
git show -s --format='merge=%H%nparents=%P' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
doneLength of output: 28932
The previous cache finding is addressed. The renamed Verified items:
You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b10d48b231
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "outcomes" | ||
| ], | ||
| "labels_per_target": 2, | ||
| "resolved_branch_or_merge_carries": "exact endpoint sets, not only the outcome", |
There was a problem hiding this comment.
When a target has multiple plausible findings on the opposite side, the label continued 1:1 does not identify which finding is the true counterpart. This protocol requires exact endpoint sets only for branch and merge labels, yet the rename table scores whether B3 selected the “exact counterpart” and natural precision likewise needs to distinguish correct from false continuations. Require every resolved continuation label to carry its exact counterpart endpoint; otherwise these correctness metrics cannot be evaluated reproducibly after truth is frozen.
Useful? React with 👍 / 👎.
| "ai_agreement_is_not_truth": "two agreeing agents produce PROPOSED truth only; the final truth artifact requires human/owner adjudication and sign-off", | ||
| "evidence_cards_contain": [ | ||
| "source at A and B", | ||
| "raw finding", |
There was a problem hiding this comment.
When evidence cards are built from the normalized findings used by this experiment, a “raw finding” includes the rule field (aggregate/normalize.py:87), contradicting the later requirement that labelers must not see rule IDs. Unless the card explicitly strips that field, both labelers receive information the blinded protocol prohibits, so the resulting truth artifact cannot satisfy the preregistered blinding conditions.
Useful? React with 👍 / 👎.
Two substantive truth-protocol defects, both found before any truth label exists. P1. A `continued 1:1` label recorded what happened but not with whom. Endpoint sets were required only for branched 1:N and merged N:1, yet the rename axis scores resolution to the exact counterpart, and natural precision must separate a correct continuation from a continuation to a neighbouring similar finding. With two plausible candidates on the far side, nobody could decide after the freeze whether B3 was right. endpoint_contract now covers every entry of the vocabulary with an explicit cardinality: continued 1:1 exactly 1 opposite-side endpoint branched 1:N >= 2 successor endpoints merged N:1 >= 2 predecessor endpoints ended empty opposite-side set new empty opposite-side set unresolved-by-truth, unadjudicable not resolved relations ended and new carry a structurally empty set on purpose. Natural row 1 fails on a fabricated new/ended where truth has a counterpart, so the scorer applies one endpoint contract instead of guessing per verdict what absence meant. Endpoint identity is a frozen hash over the canonical identity of one normalized finding, computed by the truth-artifact builder. Labelers designate endpoints by the candidate index on the evidence card and never see the preimage. Prose such as "the finding one line below" is prohibited: a natural-language foreign key cannot be scored. P1. The evidence card carried a raw finding while the protocol forbids labelers to see rule ids. This is not hypothetical. The canonical normalized finding carries rule, category, category_name, pattern_id and occurrence_id, confirmed against a real normalized snapshot from the probe. The fix is an allowlist, never a subtraction: path, line, start_line, start_column, message. "Raw finding minus rule" would silently start carrying whatever field the normalizer gains next. Forbidden fields are enumerated, and the naming gap between "rule ids" in the prohibition and `rule` in the schema is stated rather than assumed. tool is excluded because one producer is used for the whole experiment, so it is constant and carries nothing for adjudication. truth_protocol changes on purpose here, having been asserted unchanged in earlier revisions: both defects are in the truth representation itself. label_vocabulary, labels_per_target, labelers_must_not_see and ai_agreement_is_not_truth are untouched. Construction-time checks: allowlist and forbidden list disjoint, every vocabulary entry has an endpoint contract, no entry outside the vocabulary, no resolved outcome without a cardinality. Not repository-enforced; see verdict_taxonomy.verification_status. Asserted unchanged: frozen_targets, seeds, verdict_taxonomy.axes, axis_disposition, consistency_rule.scope, all four outcome tables, and truth_protocol.label_vocabulary. Manifest SHA-256 858d80beeeb82619a0adc2fb29c505d1c03831cea2344541b576a7bb0139ee3e supersedes 08f266e927d733ac177a6b5592da6cab4db51df8b82467e0bacc3b7f6b342758. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
Truth needs endpoints, and the evidence card needs an allowlistNew exact HEAD Both findings are valid, both are substantive, and both surfaced before any truth label exists — which is what preregistration is for. P1 —
Endpoint identity is a frozen hash over the canonical identity of one normalized finding, computed by the truth-artifact builder. Labelers designate endpoints by the candidate index on the card and never see the preimage — which keeps rule and identity machinery out of their view while still producing a machine-comparable record. Prose such as "the finding one line below" is prohibited; a natural-language foreign key cannot be scored. P1 — the evidence card leaked forbidden fields. Not hypothetical: Fixed with an allowlist, never a subtraction: Expected grep hit: "raw finding" still appears twice, in
Construction-time checks: allowlist and forbidden list disjoint, every vocabulary entry has an endpoint contract, none outside it, no resolved outcome without a cardinality. Not repository-enforced — see Verified from the committed blobs: zero carriage returns, Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='e418ec72076605e8c735c8971c7eff6228b34ae6'
parent='b10d48b231de80fa4a5deeb636174a555188a74b'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
echo '== Commit identity and scope =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=16 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Committed-blob checksum and encoding =='
python3 - <<'PY'
import hashlib, subprocess, tempfile
from pathlib import Path
commit='e418ec72076605e8c735c8971c7eff6228b34ae6'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
m=subprocess.check_output(['git','show',f'{commit}:{manifest}'])
s=subprocess.check_output(['git','show',f'{commit}:{sidecar}'])
print('manifest_has_cr=',b'\r' in m)
print('sidecar_has_cr=',b'\r' in s)
print('manifest_sha256=',hashlib.sha256(m).hexdigest())
print('sidecar=',repr(s))
with tempfile.TemporaryDirectory() as d:
root=Path(d)
(root/'rh-m0-manifest.json').write_bytes(m)
(root/'rh-m0-manifest.sha256').write_bytes(s)
result=subprocess.run(['sha256sum','--check','rh-m0-manifest.sha256'],
cwd=d,capture_output=True,text=True)
print('sha256sum_exit=',result.returncode)
print('sha256sum_stdout=',result.stdout.strip())
print('sha256sum_stderr=',result.stderr.strip())
PY
echo '== Changed top-level JSON surfaces =='
python3 - <<'PY'
import json, subprocess
p='research/rh-m0/rh-m0-manifest.json'
a=json.loads(subprocess.check_output(['git','show','b10d48b231de80fa4a5deeb636174a555188a74b:'+p]))
b=json.loads(subprocess.check_output(['git','show','e418ec72076605e8c735c8971c7eff6228b34ae6:'+p]))
print('changed_top_level_keys=',sorted(k for k in set(a)|set(b) if a.get(k)!=b.get(k)))
for k in ('frozen_targets','seeds','primary_holdout','hard_challenge','rename_capability','branch_capability'):
print(k+'_unchanged=',a[k]==b[k])
for k in ('axes','axis_disposition'):
print('verdict_taxonomy.'+k+'_unchanged=',a['verdict_taxonomy'][k]==b['verdict_taxonomy'][k])
PY
echo '== Targets, sample accounting, history, and side determination =='
python3 - <<'PY'
import hashlib,json,subprocess
from collections import Counter,defaultdict
p='research/rh-m0/rh-m0-manifest.json'
revs=['cde73560f029235959b7775a8466fca059c3f34e','9c0a8618f131e1bf0362d80538bb2af41f2d2c9a','6555e68944296c0f47dd5dbf52c59c76548f8422','03d56ace0623db64abe9db9d88c385717968ccca','5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa','ac54fcb3420550f0423366842fdd488f69bf1b3a','6e3304a498951089f8945f6cb3ec99978e65e887','ad58b29383d36b9fad0f0dc65a5faa3ace6636e8','11057180fe0c12011ea82c7f85cb57483c0225da','b10d48b231de80fa4a5deeb636174a555188a74b','e418ec72076605e8c735c8971c7eff6228b34ae6']
docs=[json.loads(subprocess.check_output(['git','show',f'{r}:{p}'])) for r in revs]
m=docs[-1]; targets=m['frozen_targets']['targets']
lists=[[t['target_hash'] for t in d['frozen_targets']['targets']] for d in docs]
print('revision=',m['manifest_revision']['revision'])
print('target_lists_identical_all_11=',all(x==lists[0] for x in lists[1:]))
print('counts=',m['frozen_targets']['count'],len(targets),dict(sorted(Counter(t['class'] for t in targets).items())))
pairs=defaultdict(set)
for t in targets:pairs[t['class']].add((t['a_sha'],t['b_sha']))
for c in ('natural','rename','copy'):
no={(x['a_sha'],x['b_sha']) for x in m['sample']['no_target'][c]}
print(c+'_selected_targetable_no_target_accounted=',m['sample']['selected_pairs'][c],len(pairs[c]),len(no),len(pairs[c]|no))
seed=m['seeds']['truth_seed']; bad=[]
for i,t in enumerate(targets):
if t['class']=='natural':
raw=json.dumps(['rh-m0-side/v1',seed,'sts',t['a_sha'],t['b_sha']],ensure_ascii=False,separators=(',',':')).encode()
expected='A' if hashlib.sha256(raw).digest()[0]&1==0 else 'B'
else: expected='A'
if t['side']!=expected:bad.append((i,t['class'],t['side'],expected))
print('side_mismatches=',bad)
print('truth_labels_created=',m['manifest_revision']['truth_labels_created'])
print('b3_executed=',m['manifest_revision']['b3_executed'])
for h in m['manifest_revision']['history']:
actual=hashlib.sha256(subprocess.check_output(['git','show',f"{h['commit']}:{p}"])).hexdigest()
print('history_sha',h['revision'],actual==h['sha256'])
PY
echo '== Truth endpoint and evidence-card protocol =='
python3 - <<'PY'
import json
p='research/rh-m0/rh-m0-manifest.json'
with open(p,encoding='utf-8') as f:m=json.load(f)
t=m['truth_protocol']
print(json.dumps(t,indent=2,sort_keys=True))
vocab=set(t['label_vocabulary'])
contract=t.get('endpoint_contract',{})
print('contract_labels_missing=',sorted(vocab-set(contract)))
print('contract_labels_unknown=',sorted(set(contract)-vocab))
for label,entry in contract.items():
print('endpoint_contract',repr(label),json.dumps(entry,sort_keys=True))
projection=t.get('labeler_finding_projection',{})
allow=set(projection.get('allowlist',projection.get('allowed_fields',[])))
forbidden=set(projection.get('forbidden_fields',[]))
print('projection_allowlist=',sorted(allow))
print('projection_forbidden=',sorted(forbidden))
print('allow_forbidden_overlap=',sorted(allow&forbidden))
print('labelers_must_not_see=',json.dumps(t.get('labelers_must_not_see'),sort_keys=True))
PY
echo '== Relevant normalized-finding schema references =='
rg -n -i -C 3 \
'occurrence_id|pattern_id|category_name|"rule"|"start_column"|normalized finding|normalized_finding' \
aggregate contracts research -g '*.py' -g '*.json' || trueLength of output: 172879 🏁 Script executed: #!/bin/bash
set -euo pipefail
manifest='research/rh-m0/rh-m0-manifest.json'
echo '== Exact truth-protocol section with line numbers =='
rg -n -C 8 \
'"truth_protocol"|"endpoint_contract"|"by_outcome"|"endpoint_identity"|"labeler_finding_projection"|"labels_per_target"|"truth_artifact_hashed_and_frozen_before"' \
"$manifest"
echo '== Endpoint-contract and projection invariants =='
python3 - <<'PY'
import json
from collections import Counter
p='research/rh-m0/rh-m0-manifest.json'
with open(p, encoding='utf-8') as f:
m=json.load(f)
truth=m['truth_protocol']
vocabulary=truth['label_vocabulary']
by_outcome=truth['endpoint_contract']['by_outcome']
projection=truth['labeler_finding_projection']
allowlist=projection['allowlist']
forbidden=projection['forbidden_finding_fields']
print('vocabulary_duplicates=', [x for x,n in Counter(vocabulary).items() if n > 1])
print('endpoint_contract_missing_vocabulary=', sorted(set(vocabulary)-set(by_outcome)))
print('endpoint_contract_unknown_outcomes=', sorted(set(by_outcome)-set(vocabulary)))
print('allowlist_duplicates=', [x for x,n in Counter(allowlist).items() if n > 1])
print('forbidden_duplicates=', [x for x,n in Counter(forbidden).items() if n > 1])
print('allowlist_forbidden_overlap=', sorted(set(allowlist)&set(forbidden)))
for outcome in vocabulary:
entry=by_outcome[outcome]
print('outcome=',repr(outcome),
'resolved=',entry.get('resolved'),
'endpoint_set=',repr(entry.get('endpoint_set')),
'cardinality=',repr(entry.get('cardinality')))
resolved=[name for name,entry in by_outcome.items() if entry['resolved']]
unresolved=[name for name,entry in by_outcome.items() if not entry['resolved']]
print('resolved_without_endpoint_set=', [x for x in resolved if by_outcome[x]['endpoint_set'] is None])
print('resolved_without_cardinality=', [x for x in resolved if not by_outcome[x]['cardinality']])
print('unresolved_with_endpoint_set=', [x for x in unresolved if by_outcome[x]['endpoint_set'] is not None])
identity=truth['endpoint_contract']['endpoint_identity']
print('endpoint_digest=',identity.get('digest'))
print('endpoint_preimage=',identity.get('preimage'))
print('endpoint_serialisation=',identity.get('serialisation'))
print('candidate_index_designation=',identity.get('how_labelers_designate_endpoints'))
print('free_text_prohibition=',identity.get('free_text_prohibited'))
PY
echo '== Candidate-index and truth-artifact references =='
rg -n -i -C 4 \
'candidate index|candidate_index|endpoint_id|endpoint set|truth artifact|truth_artifact|endpoint_contract' \
. -g '*.json' -g '*.py' -g '*.md' || true
echo '== Actual normalized-finding field set versus evidence projection =='
python3 - <<'PY'
import ast, json
from pathlib import Path
manifest=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
allow=set(manifest['truth_protocol']['labeler_finding_projection']['allowlist'])
forbidden=set(manifest['truth_protocol']['labeler_finding_projection']['forbidden_finding_fields'])
tree=ast.parse(Path('aggregate/normalize.py').read_text(encoding='utf-8'))
v1=None
for node in tree.body:
if isinstance(node, ast.Assign):
if any(isinstance(t,ast.Name) and t.id=='V1_FIELDS' for t in node.targets):
v1=set(ast.literal_eval(node.value))
break
if v1 is None:
raise SystemExit('V1_FIELDS not found')
identity={'pattern_id','occurrence_id','physical_anchor','identity_limitations'}
normalized=v1|identity
print('normalized_fields=',sorted(normalized))
print('allowlist_not_normalized=',sorted(allow-normalized))
print('normalized_sensitive_not_forbidden=',sorted((normalized-allow)-forbidden))
print('allowlist_sensitive=',sorted(allow&forbidden))
PYLength of output: 20837
The other requested checks passed:
You are interacting with an AI system. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e418ec7207
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "<rule>", | ||
| "<message>", | ||
| "<start_line>", | ||
| "<start_column>" |
There was a problem hiding this comment.
When two normalized findings have identical path, rule, message, line, and column, this preimage gives them the same endpoint_id; normalize_results preserves such duplicate SARIF rows, and the existing occurrence-identity contract deliberately refuses identity for this case. A branch or merge containing both rows therefore collapses to one set member, so its required >= 2 cardinality and exact-set scoring cannot be represented. Add an ambiguity census and emit a limitation/refusal rather than minting the colliding endpoint ID.
AGENTS.md reference: AGENTS.md:L10-L10
Useful? React with 👍 / 👎.
| "line", | ||
| "start_line", | ||
| "start_column", |
There was a problem hiding this comment.
When a candidate's column is needed to distinguish two findings on the same line, this allowlist cannot expose it: normalized-findings/v2 has a top-level line, but start_line and start_column are nested under physical_anchor, not top-level finding fields. Unless nested paths or explicit derived projection fields are defined, both names select nothing and same-line candidates can appear indistinguishable to labelers even though the builder hashes their columns into different endpoints.
AGENTS.md reference: AGENTS.md:L10-L10
Useful? React with 👍 / 👎.
…to real fields P1. endpoint_id was minted from a canonical preimage that two identical rows of a normalized snapshot can share. The normalizer is not obliged to deduplicate SARIF rows, so a branch relation over two physically present findings could collapse to a one-element set and silently break the >= 2 cardinality the contract promises. Fixed by refusing an identity rather than inventing one. An endpoint_id is minted only when its canonical preimage occurs exactly once in the snapshot of that revision; otherwise the id is unavailable, with the limitation endpoint-id-unavailable:ambiguous-canonical-identity. Ordinal and result-index tiebreakers are PROHIBITED: disambiguating by position would reintroduce exactly the artificial identity the occurrence contract already refused. The truth consequence is deliberately narrow. A resolved relation that would need an unavailable endpoint must not be materialized as continued, branched or merged, and its truth is unadjudicable. An ambiguous preimage elsewhere does NOT by itself spoil a target: if the truth is genuinely ended or new, the structurally empty opposite-side set is still representable and the target is adjudicated normally. Measured while constructing this revision: zero ambiguous preimages across 271 snapshots and 76253 findings of the frozen sample. The rule is preventive, not corrective. It has to be specified before truth exists rather than discovered afterwards, because the normalizer guarantees nothing here. P2. The projection allowlist named start_line and start_column, which do not exist at the top level of a normalized finding; they live inside physical_anchor, while `line` does exist at the top level. The builder was therefore still free to decide after the freeze what those two names referred to. Confirmed against a real normalized snapshot, not against the schema text. Replaced by explicit leaf-level mappings: path <- path line <- physical_anchor.start_line column <- physical_anchor.start_column message <- message One semantic coordinate per output name, and no output called start_line or start_column, so no name maps to two possible sources. physical_anchor is never admitted as a whole object: an allowlist that hands over a container has stopped deciding anything. Construction-time checks: every projection source exists in the schema, output names are unique, no source is a forbidden field or lies beneath one, and the endpoint preimage multiplicity was measured. Checks on the truth artifact itself constrain a builder that does not exist yet; only their preconditions could be measured. Not repository-enforced. Asserted unchanged: frozen_targets, seeds, verdict_taxonomy.axes, axis_disposition, all four outcome tables, truth_protocol.label_vocabulary and endpoint_contract.by_outcome. Manifest SHA-256 5a6d6746691ccac2b8e653ed29c4194d206f28bc98624252b644d3726019b3f3 supersedes 858d80beeeb82619a0adc2fb29c505d1c03831cea2344541b576a7bb0139ee3e. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
Refusing an ambiguous identity, and mapping the projection to fields that existNew exact HEAD P1 — the identity was not injective where cardinality is promised. Valid, and a good catch immediately after making truth machine-comparable. Two rows sharing every preimage field collapse to one Fixed by refusing an identity rather than inventing one:
The truth consequence is deliberately narrow. A resolved relation needing an unavailable endpoint must not be materialized as Measured while building this revision: zero ambiguous preimages across 271 snapshots and 76,253 findings of the frozen sample. The rule is therefore preventive, not corrective — which is exactly why it has to be specified before truth exists rather than discovered afterwards. P2 — the projection named fields that do not exist at that level. Valid, raised independently by both reviewers.
One semantic coordinate per output, no output named Construction-time checks: every projection source exists in the schema, output names are unique, no source is a forbidden field or lies beneath one, and endpoint preimage multiplicity was measured. The two checks that constrain the truth artifact itself bind a builder that does not exist yet; only their preconditions could be measured here. Not repository-enforced. Verified from the committed blobs: zero carriage returns, Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='ff66d0fde283e692935538b181e30602ed2073da'
parent='e418ec72076605e8c735c8971c7eff6228b34ae6'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
echo '== Commit identity and changed paths =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=20 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Committed-blob encoding and checksum =='
python3 - <<'PY'
import hashlib, subprocess, tempfile
from pathlib import Path
commit='ff66d0fde283e692935538b181e30602ed2073da'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
m=subprocess.check_output(['git','show',f'{commit}:{manifest}'])
s=subprocess.check_output(['git','show',f'{commit}:{sidecar}'])
print('manifest_has_cr=',b'\r' in m)
print('sidecar_has_cr=',b'\r' in s)
print('manifest_sha256=',hashlib.sha256(m).hexdigest())
print('sidecar=',repr(s))
with tempfile.TemporaryDirectory() as d:
root=Path(d)
(root/'rh-m0-manifest.json').write_bytes(m)
(root/'rh-m0-manifest.sha256').write_bytes(s)
r=subprocess.run(['sha256sum','--check','rh-m0-manifest.sha256'],
cwd=d,text=True,capture_output=True)
print('sha256sum_exit=',r.returncode)
print('sha256sum_stdout=',r.stdout.strip())
print('sha256sum_stderr=',r.stderr.strip())
PY
echo '== Updated truth protocol =='
python3 - <<'PY'
import json
p='research/rh-m0/rh-m0-manifest.json'
with open(p,encoding='utf-8') as f:m=json.load(f)
print(json.dumps(m['truth_protocol'],indent=2,sort_keys=True))
PY
echo '== Protocol and projection invariants =='
python3 - <<'PY'
import ast,json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
truth=m['truth_protocol']
identity=truth['endpoint_contract']['endpoint_identity']
projection=truth['labeler_finding_projection']
mappings=projection.get('mappings',projection.get('field_mappings',[]))
print('projection_mappings=',json.dumps(mappings,sort_keys=True))
outputs=[x.get('output') for x in mappings]
sources=[x.get('source') for x in mappings]
print('projection_outputs_unique=',len(outputs)==len(set(outputs)))
print('projection_sources=',sources)
print('whole_physical_anchor_admitted=',any(x=='physical_anchor' for x in sources))
forbidden=set(projection['forbidden_finding_fields'])
print('projection_source_forbidden_or_below=',[
x for x in sources if x in forbidden or any(x.startswith(f+'.') for f in forbidden)
])
print('endpoint_identity=',json.dumps(identity,indent=2,sort_keys=True))
print('ordinal_or_result_index_allowed=',[
(k,v) for k,v in identity.items()
if 'ordinal' in k.lower() or 'result' in k.lower()
])
print('endpoint_unavailable_rule=',json.dumps(
truth['endpoint_contract'].get('unavailable_endpoint_handling',{}),sort_keys=True))
tree=ast.parse(Path('aggregate/normalize.py').read_text(encoding='utf-8'))
v1=None
for node in tree.body:
if isinstance(node,ast.Assign) and any(isinstance(t,ast.Name) and t.id=='V1_FIELDS' for t in node.targets):
v1=set(ast.literal_eval(node.value))
break
if v1 is None: raise SystemExit('V1_FIELDS not found')
schema=set(v1)|{'physical_anchor.start_line','physical_anchor.start_column',
'pattern_id','occurrence_id','identity_limitations'}
print('projection_sources_not_in_schema=',sorted(set(sources)-schema))
PY
echo '== Frozen targets, samples, and revision preservation =='
python3 - <<'PY'
import hashlib,json,subprocess
from collections import Counter,defaultdict
p='research/rh-m0/rh-m0-manifest.json'
revs=['cde73560f029235959b7775a8466fca059c3f34e','9c0a8618f131e1bf0362d80538bb2af41f2d2c9a','6555e68944296c0f47dd5dbf52c59c76548f8422','03d56ace0623db64abe9db9d88c385717968ccca','5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa','ac54fcb3420550f0423366842fdd488f69bf1b3a','6e3304a498951089f8945f6cb3ec99978e65e887','ad58b29383d36b9fad0f0dc65a5faa3ace6636e8','11057180fe0c12011ea82c7f85cb57483c0225da','b10d48b231de80fa4a5deeb636174a555188a74b','e418ec72076605e8c735c8971c7eff6228b34ae6','ff66d0fde283e692935538b181e30602ed2073da']
docs=[json.loads(subprocess.check_output(['git','show',f'{r}:{p}'])) for r in revs]
before,m=docs[-2],docs[-1]
targets=m['frozen_targets']['targets']
lists=[[x['target_hash'] for x in d['frozen_targets']['targets']] for d in docs]
print('revision=',m['manifest_revision']['revision'])
print('target_lists_identical_all_12=',all(x==lists[0] for x in lists[1:]))
print('target_counts=',m['frozen_targets']['count'],len(targets),dict(sorted(Counter(x['class'] for x in targets).items())))
pairs=defaultdict(set)
for t in targets:pairs[t['class']].add((t['a_sha'],t['b_sha']))
for cls in ('natural','rename','copy'):
no={(x['a_sha'],x['b_sha']) for x in m['sample']['no_target'][cls]}
print(cls+'_selected_targetable_no_target_accounted=',m['sample']['selected_pairs'][cls],len(pairs[cls]),len(no),len(pairs[cls]|no))
seed=m['seeds']['truth_seed']; bad=[]
for i,t in enumerate(targets):
if t['class']=='natural':
raw=json.dumps(['rh-m0-side/v1',seed,'sts',t['a_sha'],t['b_sha']],ensure_ascii=False,separators=(',',':')).encode()
expected='A' if hashlib.sha256(raw).digest()[0]&1==0 else 'B'
else: expected='A'
if t['side']!=expected:bad.append((i,t['class'],t['side'],expected))
print('side_errors=',bad)
for key in ('seeds','frozen_targets','primary_holdout','hard_challenge','rename_capability','branch_capability'):
print(key+'_unchanged_from_rev11=',before[key]==m[key])
print('truth_labels_created=',m['manifest_revision']['truth_labels_created'])
print('b3_executed=',m['manifest_revision']['b3_executed'])
for h in m['manifest_revision']['history']:
raw=subprocess.check_output(['git','show',f"{h['commit']}:{p}"])
print('history_sha',h['revision'],hashlib.sha256(raw).hexdigest()==h['sha256'])
PYLength of output: 39985 🏁 Script executed: #!/bin/bash
set -euo pipefail
manifest='research/rh-m0/rh-m0-manifest.json'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Candidate-index and truth-artifact references =='
rg -n -i -C 5 \
'candidate index|candidate_index|endpoint_id|endpoint set|endpoint_identity|truth artifact|truth_artifact|labeler_finding_projection' \
"$manifest" aggregate contracts research -g '*.json' -g '*.py' -g '*.md' || true
echo '== Actual projection mapping validation =='
python3 - <<'PY'
import ast,json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
projection=m['truth_protocol']['labeler_finding_projection']
fields=projection['fields']
outputs=[x['output'] for x in fields]
sources=[x['source'] for x in fields]
forbidden=set(projection['forbidden_finding_fields'])
print('field_count=',len(fields))
print('outputs=',outputs)
print('outputs_unique=',len(outputs)==len(set(outputs)))
print('sources=',sources)
print('whole_physical_anchor_admitted=',any(x=='physical_anchor' for x in sources))
print('sources_forbidden_or_below=',[
s for s in sources if s in forbidden or any(s.startswith(f+'.') for f in forbidden)
])
tree=ast.parse(Path('aggregate/normalize.py').read_text(encoding='utf-8'))
v1=None
for node in tree.body:
if isinstance(node,ast.Assign) and any(isinstance(t,ast.Name) and t.id=='V1_FIELDS' for t in node.targets):
v1=set(ast.literal_eval(node.value))
break
if v1 is None: raise SystemExit('V1_FIELDS not found')
valid_paths=v1|{'physical_anchor.start_line','physical_anchor.start_column',
'pattern_id','occurrence_id','identity_limitations'}
print('sources_not_in_normalized_schema=',sorted(set(sources)-valid_paths))
PY
echo '== Candidate-index contract shape =='
python3 - <<'PY'
import json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
truth=m['truth_protocol']
identity=truth['endpoint_contract']['endpoint_identity']
text=json.dumps(truth,sort_keys=True).lower()
for term in ('candidate index','candidate_index','namespace','side','revision','resolver','endpoint_id'):
print(term.replace(' ','_')+'_occurrences=',text.count(term))
print('designation=',identity['how_labelers_designate_endpoints'])
print('all_endpoint_identity_keys=',sorted(identity))
print('evidence_card_fields=',[x['output'] for x in truth['labeler_finding_projection']['fields']])
PY
echo '== Frozen Step-1 provenance =='
git cat-file -e "${merge}^{commit}"
git cat-file -e "${reviewed}^{commit}"
git show -s --format='merge=%H%nparents=%P%nsubject=%s' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
doneLength of output: 33497
The remaining checks passed:
You are interacting with an AI system. |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Revision 12 closed guessing at the level of endpoint identity and left it one level above, in the reference a human uses to point at that identity. A bare "candidate 3" has no namespace: candidate 3 can exist on side A and on side B, and on two different cards, so after the labels were written the builder would still be choosing which one was meant. candidate_ref is now the frozen reference: target_hash the frozen target whose card this is revision_role A or B revision_sha the exact commit of that side candidate_index non-negative integer assigned by the card The full four-part key must be unique across the evidence corpus, and role and sha cross-check: role A requires the target's a_sha, role B its b_sha. A:1 and B:1 become trivially distinguishable, and a mistyped sha is caught rather than guessed. candidate_index is presentation-local. It must not participate in endpoint_id, in the canonical preimage, in ambiguity resolution, or in any tiebreaker. Otherwise the SARIF ordinal, prohibited one revision ago, walks back in through the front door under a new name. The index addresses a row on a card; it never identifies a finding. candidate_endpoint_mapping resolves candidate_ref to endpoint_id or unavailable. It is built during evidence-card construction, before any labeler response is interpreted and before any label exists, and is persisted and hashed with the truth artifact. Truth stores both candidate_ref and resolved_endpoint_id: the scorer needs a machine-comparable identity, and a later reader needs to see what the human was actually looking at. A candidate_ref resolving to unavailable must not be stored as an endpoint of a resolved relation, reusing the revision-12 fail-closed semantics rather than introducing a second set. The labeler never sees endpoint_id, so the hash cannot become a side channel back to rule. Deliberately not done: no canonical candidate ordering is frozen. The finding was a missing foreign key, not a missing ordering, and the meaning of an index comes from the frozen mapping rather than from a sort algorithm. Freezing an ordering too would grow a second identity scheme out of one integer. Endpoint identity is untouched: mint iff the canonical preimage is unique, otherwise unavailable, ordinal tiebreakers prohibited, narrow unavailable semantics. Construction-time checks: all 63 targets carry both a_sha and b_sha so the role-to-sha rule is checkable, all 63 target_hash values are unique so candidate_ref keys cannot collide across targets, candidate_index appears nowhere in the endpoint preimage, and the role vocabulary matches the sides present in the frozen targets. Asserted unchanged: the endpoint preimage, minting rule, ordinal prohibition, endpoint_contract.by_outcome, unavailable_endpoint_semantics, label_vocabulary, labeler_finding_projection.fields, frozen_targets, seeds, verdict_taxonomy.axes and all four outcome tables. Manifest SHA-256 3a99631292122919b1dc5d9750d899da0ecde23d9ba452c731da6c29209730b4 supersedes 5a6d6746691ccac2b8e653ed29c4194d206f28bc98624252b644d3726019b3f3. No B3 run. No truth labels. No correctness result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV
The labeler's reference now has a namespaceNew exact HEAD Valid. Revision 12 closed guessing at endpoint identity and left it one level above, in the reference a human uses to point at that identity. A bare "candidate 3" has no namespace: it can exist on side A and on side B, and on two different cards, so after the labels were written the builder would still be choosing which one was meant.
The full four-part key must be unique across the evidence corpus, and role and sha cross-check: role
A Deliberately not done: no canonical candidate ordering is frozen. The finding was a missing foreign key, not a missing ordering, and the meaning of an index comes from the frozen mapping rather than from a sort algorithm. Freezing an ordering as well would grow a second identity scheme out of one integer. Endpoint identity is untouched and asserted so: preimage, minting rule, ordinal prohibition, Construction-time checks: all 63 targets carry both Verified from the committed blobs: zero carriage returns, Manifest SHA-256 No B3 run. No truth labels. No correctness result. |
|
@coderabbitai review Final review requested on new exact HEAD |
|
@codex review Final review requested on new exact HEAD |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='493000deb1149135437442e58add9a1107f2323d'
parent='ff66d0fde283e692935538b181e30602ed2073da'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
merge='011c1362861f6b8b20c45e8ebd5bcb912401c1c0'
reviewed='6dcc02f7d82bebeb5db9be83f77ab8c5455f5692'
echo '== Requested commit identity and changed paths =='
test "$(git rev-parse HEAD)" = "$head"
test "$(git rev-parse "$head^")" = "$parent"
git show -s --format='commit=%H%nparent=%P%nsubject=%s' "$head"
git diff-tree --no-commit-id --name-status -r "$head"
git diff --no-ext-diff --unified=18 "$parent" "$head" -- "$manifest" "$sidecar"
echo '== Committed-blob checksum and encoding =='
python3 - <<'PY'
import hashlib, subprocess, tempfile
from pathlib import Path
commit='493000deb1149135437442e58add9a1107f2323d'
manifest='research/rh-m0/rh-m0-manifest.json'
sidecar='research/rh-m0/rh-m0-manifest.sha256'
m=subprocess.check_output(['git','show',f'{commit}:{manifest}'])
s=subprocess.check_output(['git','show',f'{commit}:{sidecar}'])
print('manifest_has_cr=',b'\r' in m)
print('sidecar_has_cr=',b'\r' in s)
print('manifest_sha256=',hashlib.sha256(m).hexdigest())
print('sidecar_bytes=',repr(s))
with tempfile.TemporaryDirectory() as d:
root=Path(d)
(root/'rh-m0-manifest.json').write_bytes(m)
(root/'rh-m0-manifest.sha256').write_bytes(s)
r=subprocess.run(['sha256sum','--check','rh-m0-manifest.sha256'],
cwd=d,text=True,capture_output=True)
print('sha256sum_exit=',r.returncode)
print('sha256sum_stdout=',r.stdout.strip())
print('sha256sum_stderr=',r.stderr.strip())
PY
echo '== Candidate-reference and endpoint contract =='
python3 - <<'PY'
import json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
truth=m['truth_protocol']
endpoint=truth['endpoint_contract']
identity=endpoint['endpoint_identity']
for key in ('candidate_reference','candidate_ref','candidate_endpoint_mapping',
'truth_artifact','endpoint_identity'):
if key in endpoint:
print(f'endpoint_contract.{key}=',json.dumps(endpoint[key],indent=2,sort_keys=True))
if key in truth:
print(f'truth_protocol.{key}=',json.dumps(truth[key],indent=2,sort_keys=True))
print('endpoint_identity_keys=',sorted(identity))
print('endpoint_identity=',json.dumps(identity,indent=2,sort_keys=True))
print('truth_protocol_keys=',sorted(truth))
# Locate all structured candidate-reference objects without relying on prose.
def walk(value,path='$'):
if isinstance(value,dict):
keys=set(value)
if {'target_hash','revision_role','revision_sha','candidate_index'} <= keys:
print('candidate_ref_shape_at=',path,'keys=',sorted(keys))
print('candidate_ref_object=',json.dumps(value,sort_keys=True))
for k,v in value.items():
walk(v,path+'.'+k)
elif isinstance(value,list):
for i,v in enumerate(value):
walk(v,f'{path}[{i}]')
walk(truth)
PY
echo '== Candidate-reference semantic checks against frozen targets =='
python3 - <<'PY'
import json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
truth=m['truth_protocol']
targets=m['frozen_targets']['targets']
target_by_hash={t['target_hash']:t for t in targets}
# Print every value structurally associated with candidate refs/mappings.
def collect(v,path='$',out=None):
out=[] if out is None else out
if isinstance(v,dict):
for k,x in v.items():
p=path+'.'+k
if any(word in k.lower() for word in ('candidate','mapping','artifact','reference')):
out.append((p,x))
collect(x,p,out)
elif isinstance(v,list):
for i,x in enumerate(v): collect(x,f'{path}[{i}]',out)
return out
for path,value in collect(truth):
print(path,'=',json.dumps(value,sort_keys=True))
print('unique_target_hashes=',len(target_by_hash)==len(targets))
print('target_missing_a_or_b=',[
t['target_hash'] for t in targets if not t.get('a_sha') or not t.get('b_sha')
])
print('target_sides=',sorted({t['side'] for t in targets}))
PY
echo '== Frozen targets, sample accounting, side rule, and history =='
python3 - <<'PY'
import hashlib,json,subprocess
from collections import Counter,defaultdict
p='research/rh-m0/rh-m0-manifest.json'
revs=[
'cde73560f029235959b7775a8466fca059c3f34e',
'9c0a8618f131e1bf0362d80538bb2af41f2d2c9a',
'6555e68944296c0f47dd5dbf52c59c76548f8422',
'03d56ace0623db64abe9db9d88c385717968ccca',
'5c89ba515da4e5e84b3fb7749ccdf9d99bd28ffa',
'ac54fcb3420550f0423366842fdd488f69bf1b3a',
'6e3304a498951089f8945f6cb3ec99978e65e887',
'ad58b29383d36b9fad0f0dc65a5faa3ace6636e8',
'11057180fe0c12011ea82c7f85cb57483c0225da',
'b10d48b231de80fa4a5deeb636174a555188a74b',
'e418ec72076605e8c735c8971c7eff6228b34ae6',
'ff66d0fde283e692935538b181e30602ed2073da',
'493000deb1149135437442e58add9a1107f2323d',
]
docs=[json.loads(subprocess.check_output(['git','show',f'{r}:{p}'])) for r in revs]
before,m=docs[-2],docs[-1]
targets=m['frozen_targets']['targets']
lists=[[t['target_hash'] for t in d['frozen_targets']['targets']] for d in docs]
print('revision=',m['manifest_revision']['revision'])
print('target_lists_identical_all_13=',all(x==lists[0] for x in lists[1:]))
print('target_counts=',m['frozen_targets']['count'],len(targets),dict(sorted(Counter(t['class'] for t in targets).items())))
print('unique_valid_target_hashes=',len(set(lists[-1]))==len(lists[-1]) and all(len(h)==64 and set(h)<=set('0123456789abcdef') for h in lists[-1]))
pairs=defaultdict(set)
for t in targets:pairs[t['class']].add((t['a_sha'],t['b_sha']))
for cls in ('natural','rename','copy'):
no={(x['a_sha'],x['b_sha']) for x in m['sample']['no_target'][cls]}
print(cls+'_selected_targetable_no_target_accounted=',m['sample']['selected_pairs'][cls],len(pairs[cls]),len(no),len(pairs[cls]|no))
seed=m['seeds']['truth_seed']; bad=[]
for i,t in enumerate(targets):
if t['class']=='natural':
raw=json.dumps(['rh-m0-side/v1',seed,'sts',t['a_sha'],t['b_sha']],ensure_ascii=False,separators=(',',':')).encode()
expected='A' if hashlib.sha256(raw).digest()[0]&1==0 else 'B'
else:
expected='A'
if t['side']!=expected: bad.append((i,t['class'],t['side'],expected))
print('side_rule_errors=',bad)
print('truth_labels_created=',m['manifest_revision']['truth_labels_created'])
print('b3_executed=',m['manifest_revision']['b3_executed'])
for key in ('seeds','frozen_targets','primary_holdout','hard_challenge','rename_capability','branch_capability'):
print(key+'_unchanged_from_rev12=',before[key]==m[key])
for key in ('axes','axis_disposition'):
print('taxonomy_'+key+'_unchanged_from_rev12=',before['verdict_taxonomy'][key]==m['verdict_taxonomy'][key])
for e in m['manifest_revision']['history']:
raw=subprocess.check_output(['git','show',f"{e['commit']}:{p}"])
print('history_sha_'+str(e['revision'])+'=',hashlib.sha256(raw).hexdigest()==e['sha256'])
PY
echo '== Truth vocabulary, outcome tables, and Step-1 provenance =='
python3 - <<'PY'
import json
from pathlib import Path
m=json.loads(Path('research/rh-m0/rh-m0-manifest.json').read_text(encoding='utf-8'))
truth=m['truth_protocol']; tax=m['verdict_taxonomy']
vocab=set(truth['label_vocabulary'])
contract=truth['endpoint_contract']['by_outcome']
print('endpoint_contract_missing_vocab=',sorted(vocab-set(contract)))
print('endpoint_contract_unknown_outcomes=',sorted(set(contract)-vocab))
for name in ('rename_capability','branch_capability'):
print(name+'_unknown_truth_refs=',sorted(set(m[name].get('truth_labels_referenced',[]))-vocab))
for axis,section in (('natural','primary_holdout'),('hard','hard_challenge'),('rename','rename_capability'),('branch','branch_capability')):
emitted={row['verdict'] for row in m[section]['outcome_table']}
declared=set(tax['axes'][axis])
print(axis+'_emitted_not_declared=',sorted(emitted-declared))
print(axis+'_declared_unreachable=',sorted(declared-emitted))
PY
git show -s --format='merge=%H%nparents=%P' "$merge"
git merge-base --is-ancestor "$reviewed" "$merge" && echo 'reviewed_head_is_ancestor=true'
for path in contracts/finding-lineage-decision-v1.json contracts/finding-lineage-v1.json; do
printf '%s merge_blob=%s reviewed_blob=%s\n' "$path" \
"$(git rev-parse "$merge:$path")" "$(git rev-parse "$reviewed:$path")"
doneLength of output: 41388
The previous candidate-reference blocker is addressed.
I also verified these manifest invariants:
You are interacting with an AI system. |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Freezes the RH-M0 manifest for the finding-lineage external-validity experiment.
No B3 run has happened. No truth label exists. No correctness result is in this PR.
Current state
3a99631292122919b1dc5d9750d899da0ecde23d9ba452c731da6c29209730b4011c1362861f6b8b20c45e8ebd5bcb912401c1c06dcc02f7d82bebeb5db9be83f77ab8c5455f5692The 63 frozen target hashes are byte-identical across every revision of this PR.
Only the specification surface changed.
Revision history
b22cb93d…cde73560df0e9f8d…9c0a8618b113a318…6555e6892b8ce701…03d56acef4c5181e…5c89ba517d087e8a…ac54fcb371f4c923…6e3304a4ff6a5ed4…ad58b293502477c4…1105718008f266e9…b10d48b2858d80be…e418ec725a6d6746…ff66d0fd3a996312…Corrections carried in
Identical
pattern_idmultiset does not mean a trivial pair.pattern_idissha1(path + rule + message)and excludes the line, so an identical multiset iscompatible with every occurrence having moved. Such a pair is a direct line-drift
witness. The natural sample is not resampled.
The old hard-transform criterion was unsatisfiable. It required B3 to have
fewer unsafe relation errors than B1/B2. Both baselines key on
same_pattern_id,which includes the path, so across a rename or copy they answer
unresolved— notan unsafe error. Their unsafe-error count is naturally zero. Replaced by a
safe-added-resolution criterion.
Side determination is per class. Rename and copy candidates must be A-side
findings on the transform source path, so their side is forced by construction and
the side hash is never consulted. Applying the natural rule to the hard classes
would yield side B for 6 of the 13 hard targets and those target hashes would not
reproduce.
seeds.side_determinationstates the rule per class and carries areproduction warning.
Every constraint is a prohibition that is true, never a permission that is
false. No reader has to decide what a denied permission means.
No axis may be passed without resolving anything. Zero unsafe errors is
trivially achieved by answering
unresolvedeverywhere; safety withoutresolution is not a capability. Every axis carries a resolution floor, and the
rename and branch outcome tables are exhaustive and precedence-ordered.
NO_TARGET is kept, not repaired
17 of 21 rename pairs and 20 of 25 copy pairs yield no target: findings cover
roughly 199 of ~1900
.csfiles, and the intersection with rename and copy sourcesis small. Those pairs stay in the frame as evidence of sparsity. No pair was
replaced.
Ceiling
merged_symbolshas no observation source, and none was invented for thebenchmark. MERGE-INCONCLUSIVE is fixed for this experiment.
🤖 Generated with Claude Code
https://claude.ai/code/session_015vtUmvBDf69ccY5ju3PSHV