Skip to content

Commit f09fa00

Browse files
claudeclaude
committed
feat(perf): #263-A — the measurement instrument, calibrated and firewalled
The instrument for #263's P-022 performance baselines: how a number is produced, never what the number should be. It permits calibration and refuses the decisive measurement, which is #263-B behind the D7 freeze. No thresholds, no repetition count N, no engine-comparison statistic — those are D7's, and a harness that offered any of them would be choosing them. Preflight, against primary source. Live #263 and #262's Performance-gates reconcile as: the brief's phase list is #262's seven gate phases plus frontend extraction, recorded and explicitly NOT D7-gated. Both populations are frozen — A, the G3 cutover subset, feeds D7; B, the IDE-foundation remainder, does not unless separately ratified. What this instrument does NOT measure is written down rather than left implied, so completing #263-A cannot narrow #263's own acceptance by omission: the .own workloads, OwnIR serialization, no-op and single-file-edit recheck, the latency proxy, allocation profiles and IDE budgets are owed on B's track. The honest limit, recorded rather than routed around. Neither engine exposes per-phase timing through its production surface and #263-A may not add production instrumentation, so phases are measured by a LADDER of real production invocations, each recorded as the composed interval it actually is with the phases inside it named. bridge-lowering and analysis are marked NOT SEPARATELY OBSERVABLE. Derived views — parse by subtraction, core work as end-to-end minus extraction — are labelled derived and never presented as measurements. #263 asks for unavailable stages to be marked, not imputed. The firewall is structural, not clerical. Exactly one function starts a clock and it refuses a decisive workload unless the identity gate is armed, which needs a D7 attestation that does not exist. Deliberately stricter than "no paired runs": a single-engine timed run of a decisive workload is refused too, because a stopwatch appears nowhere in the permitted list of enumerate / hash / fetch / availability-check / non-timed smoke. "We did not save the delta" is accounting-clean and epistemically worthless — whoever watched the two numbers has already peeked. The four carried obligations live in the harness. The identity gate exists NOW and dormant, because D7's C1 freezes the harness digest and a gate added after that freeze makes this a different instrument. Arming is a JSON file: a control proves the digest does not move when the gate arms and that the firewall then opens. Verification completes before any clock starts, and a control counts digest calls INSIDE a measured interval and requires zero — a gate that pays for itself out of the startup benchmark is a defect wearing a safeguard's coat. The candidate binary is a separate domain: frozen at session start, its drift refuses the remainder rather than measuring half the cells on another binary. Peak RSS comes from os.wait4's per-child ru_maxrss rather than /usr/bin/time, which is a package that can simply be absent — it is absent here, and "the tool was missing" is not a memory measurement. Windows uses a Job Object. Where nothing is available the value is null WITH A REASON, never a confident zero. Two defects of my own, found while building this and worth recording. The comparison guard first grepped the source for "ratio" and refused its own report — the sentence explaining the prohibition matched it, and then so did "calibration" and "iterations". It now reads the report's structure and matches whole tokens. And an aggregate of bytes was being emitted under a median_ns key: a unit lie in a report whose entire purpose is measurement. Calibrated: 44 cells, both engines, all six rungs, run twice on one machine with every cell's median reproducing within the noise floor that run itself recorded. CI calibrates on Linux and Windows and asserts the same, plus that no decisive workload was timed and the gate stayed dormant. scripts/perf_baseline.py is a new wrapper that reaches the launcher, so it is added to the Stage-2 census entry points and the job classified — the census enumerates wrappers, which makes that list a maintenance obligation, now stated in the ledger. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb
1 parent ca8b106 commit f09fa00

6 files changed

Lines changed: 1971 additions & 2 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2894,6 +2894,95 @@ jobs:
28942894
[ "$rc" -ne 0 ] || { echo "FAIL: a broken candidate exited 0"; exit 1; }
28952895
echo "OK: a broken candidate is a visible failure (exit $rc), never a Python rescue"
28962896
2897+
# #263-A — the performance measurement INSTRUMENT, calibrated. Not a
2898+
# baseline: this job proves the instrument stands up, on both platforms, and
2899+
# that it refuses to measure anything decisive before the D7 freeze.
2900+
#
2901+
# It publishes no engine comparison and can publish none: the harness holds
2902+
# one engine's samples per cell and refuses a report carrying a comparison
2903+
# key. What it asserts here is that the refusal is real, not that Rust is
2904+
# anything.
2905+
perf-instrument:
2906+
name: "#263-A measurement instrument (CALIBRATION_ONLY)"
2907+
strategy:
2908+
fail-fast: false
2909+
matrix:
2910+
os: [ubuntu-latest, windows-latest]
2911+
runs-on: ${{ matrix.os }}
2912+
defaults:
2913+
run:
2914+
shell: bash
2915+
steps:
2916+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
2917+
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4
2918+
with:
2919+
dotnet-version: "8.0.x"
2920+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
2921+
with:
2922+
python-version: "3.13"
2923+
- uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10
2924+
with:
2925+
toolchain: stable
2926+
- name: Build the production own-cli candidate
2927+
working-directory: rust
2928+
run: cargo build -p own-cli --release
2929+
- name: The instrument's own selftest (no measurement taken)
2930+
run: python scripts/perf_baseline.py --selftest
2931+
- name: The instrument controls (firewall, both identity domains, provenance)
2932+
run: python tests/test_perf_instrument.py
2933+
# Decisive workloads: availability and pins only. No clock, no resource
2934+
# sampling — this step is what "enumerated, hashed, availability-checked"
2935+
# looks like when it is the only thing permitted.
2936+
- name: Decisive workloads — non-timed structural smoke
2937+
run: python scripts/perf_baseline.py --smoke-decisive
2938+
- name: Calibrate, twice, and check the run reproduces itself
2939+
run: |
2940+
ext=""
2941+
if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi
2942+
export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli$ext"
2943+
python scripts/perf_baseline.py --calibrate --repeat 5 --warmup 2 \
2944+
--out "$RUNNER_TEMP/cal-1.json"
2945+
python scripts/perf_baseline.py --calibrate --repeat 5 --warmup 2 \
2946+
--reproduce "$RUNNER_TEMP/cal-1.json" --out "$RUNNER_TEMP/cal-2.json"
2947+
- name: The calibration holds — and measured nothing decisive
2948+
run: |
2949+
python - "$RUNNER_TEMP/cal-2.json" <<'PY'
2950+
import json, sys
2951+
rep = json.load(open(sys.argv[1], encoding="utf-8"))
2952+
man = json.load(open("docs/evidence/p022-263a-workloads.json", encoding="utf-8"))
2953+
decisive = {w["id"] for w in man["decisive"]}
2954+
problems = []
2955+
if rep.get("tag") != "CALIBRATION_ONLY":
2956+
problems.append("the report is not tagged CALIBRATION_ONLY")
2957+
if rep["identity"]["gate_armed"]:
2958+
problems.append("the D7 gate is ARMED — #263-A must never ship an attestation")
2959+
timed = sorted({c["workload"] for c in rep["cells"]} & decisive)
2960+
if timed:
2961+
problems.append(f"DECISIVE workloads were timed: {timed}")
2962+
if rep["noise"]["invalidated"]:
2963+
problems.append(f"the run was invalidated: {rep['noise']['invalidation_reasons']}")
2964+
if not rep["reproducibility"]["reproduced"]:
2965+
problems.append(f"the run did not reproduce: "
2966+
f"{rep['reproducibility']['cells_outside_tolerance']}")
2967+
if not all(c.get("raw_elapsed_ns") for c in rep["cells"]):
2968+
problems.append("a cell kept no raw per-iteration data")
2969+
rss = {c["rss_mechanism"] for c in rep["cells"]}
2970+
if rss == {"none"}:
2971+
problems.append("no named RSS mechanism was available, so §8 was skipped entirely")
2972+
print("\n".join(problems) if problems else
2973+
f"instrument OK: {len(rep['cells'])} calibration cells, "
2974+
f"{rep['reproducibility']['compared_cells']} reproduced within "
2975+
f"{rep['reproducibility']['tolerance_relative']}, RSS via {sorted(rss)}, "
2976+
f"gate dormant, 0 decisive workloads timed")
2977+
sys.exit(1 if problems else 0)
2978+
PY
2979+
- name: Upload the calibration reports
2980+
if: always()
2981+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
2982+
with:
2983+
name: perf-calibration-${{ matrix.os }}
2984+
path: ${{ runner.temp }}/cal-*.json
2985+
28972986
# P-014 Tier B: external-reference resolution. The SAME sample, run two ways, must give two
28982987
# verdicts — proving the extractor binds a THIRD-PARTY event only when its DLL is referenced:
28992988
# A (no refs) -> ObservableObject is an error type -> OWN050 (honest skip), no leak

‎docs/evidence/p022-263a-workloads.json‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
{
2+
"schema": 1,
3+
"comment": "#263-A FROZEN workload manifest. Two populations that must never be confused: DECISIVE workloads feed the D7 cutover verdict and MUST NOT be run in timed or resource-measured mode before the D7 freeze; CALIBRATION workloads exist only to validate the instrument and can never pick a threshold. The split is why calibration can happen at all without anybody having peeked at the Rust-vs-Python delta — 'we did not save it' is accounting-clean, not epistemically clean.",
4+
"populations": {
5+
"decisive": "The G3 cutover subset (#262's Performance gates) plus #262's large-solution control. Pinned by commit; drift is a FAILED TARGET, never a newer measurement. Pre-D7 these may be enumerated, hashed, fetched, availability-checked and exercised by NON-TIMED structural smoke only.",
6+
"calibration": "Generated deterministically by the harness from this manifest, so they are separate from the decisive corpus by construction rather than by discipline. Every result over them is tagged CALIBRATION_ONLY."
7+
},
8+
"decisive": [
9+
{
10+
"id": "repo-samples",
11+
"kind": "source-tree",
12+
"path": "frontend/roslyn/samples",
13+
"why": "the committed C# sample tree — the small end of the decisive size ladder"
14+
},
15+
{
16+
"id": "repo-examples",
17+
"kind": "source-tree",
18+
"path": "examples",
19+
"why": "the committed examples tree — the medium rung"
20+
},
21+
{
22+
"id": "committed-corpus",
23+
"kind": "source-tree",
24+
"path": "corpus",
25+
"why": "#263's 'corpus batch' and #262's committed-corpus gate"
26+
},
27+
{
28+
"id": "oss-ShareX.repo",
29+
"kind": "source-tree",
30+
"path": "targets/ShareX",
31+
"pin": {
32+
"repository": "ShareX/ShareX",
33+
"commit": "0df9ca4d83eed9d2489048c539d7d1fc2860fdec"
34+
},
35+
"why": "one of the five pinned OSS repositories of #243, whole-tree walk, at its verified pin"
36+
},
37+
{
38+
"id": "oss-ShareX.sln",
39+
"kind": "source-tree",
40+
"path": "targets/ShareX/ShareX.sln",
41+
"pin": {
42+
"repository": "ShareX/ShareX",
43+
"commit": "0df9ca4d83eed9d2489048c539d7d1fc2860fdec"
44+
},
45+
"why": "the largest solution of ShareX — a different extractor path and a differently ORDERED document, not a subset of the repo walk"
46+
},
47+
{
48+
"id": "oss-MahApps.Metro.repo",
49+
"kind": "source-tree",
50+
"path": "targets/MahApps.Metro",
51+
"pin": {
52+
"repository": "MahApps/MahApps.Metro",
53+
"commit": "72099e310bac2d12ac98fd7560b69679252519f5"
54+
},
55+
"why": "one of the five pinned OSS repositories of #243, whole-tree walk, at its verified pin"
56+
},
57+
{
58+
"id": "oss-MahApps.Metro.sln",
59+
"kind": "source-tree",
60+
"path": "targets/MahApps.Metro/src/MahApps.Metro.sln",
61+
"pin": {
62+
"repository": "MahApps/MahApps.Metro",
63+
"commit": "72099e310bac2d12ac98fd7560b69679252519f5"
64+
},
65+
"why": "the largest solution of MahApps.Metro — a different extractor path and a differently ORDERED document, not a subset of the repo walk"
66+
},
67+
{
68+
"id": "oss-MaterialDesignInXamlToolkit.repo",
69+
"kind": "source-tree",
70+
"path": "targets/MaterialDesignInXamlToolkit",
71+
"pin": {
72+
"repository": "MaterialDesignInXAML/MaterialDesignInXamlToolkit",
73+
"commit": "ef3a5ea434e39182b1848f5e11aaea6b3890581f"
74+
},
75+
"why": "one of the five pinned OSS repositories of #243, whole-tree walk, at its verified pin"
76+
},
77+
{
78+
"id": "oss-AvalonEdit.repo",
79+
"kind": "source-tree",
80+
"path": "targets/AvalonEdit",
81+
"pin": {
82+
"repository": "icsharpcode/AvalonEdit",
83+
"commit": "ed0bd149059469ac9bd39b13cf8a341b12a6c1da"
84+
},
85+
"why": "one of the five pinned OSS repositories of #243, whole-tree walk, at its verified pin"
86+
},
87+
{
88+
"id": "oss-AvalonEdit.sln",
89+
"kind": "source-tree",
90+
"path": "targets/AvalonEdit/ICSharpCode.AvalonEdit.sln",
91+
"pin": {
92+
"repository": "icsharpcode/AvalonEdit",
93+
"commit": "ed0bd149059469ac9bd39b13cf8a341b12a6c1da"
94+
},
95+
"why": "the largest solution of AvalonEdit — a different extractor path and a differently ORDERED document, not a subset of the repo walk"
96+
},
97+
{
98+
"id": "oss-ClosedXML.repo",
99+
"kind": "source-tree",
100+
"path": "targets/ClosedXML",
101+
"pin": {
102+
"repository": "ClosedXML/ClosedXML",
103+
"commit": "4e89dcedd83cad553e84d2d97f77fc3d7deb630f"
104+
},
105+
"why": "one of the five pinned OSS repositories of #243, whole-tree walk, at its verified pin"
106+
},
107+
{
108+
"id": "oss-ClosedXML.sln",
109+
"kind": "source-tree",
110+
"path": "targets/ClosedXML/ClosedXML.sln",
111+
"pin": {
112+
"repository": "ClosedXML/ClosedXML",
113+
"commit": "4e89dcedd83cad553e84d2d97f77fc3d7deb630f"
114+
},
115+
"why": "the largest solution of ClosedXML — a different extractor path and a differently ORDERED document, not a subset of the repo walk"
116+
},
117+
{
118+
"id": "large-solution-control",
119+
"kind": "source-tree",
120+
"path": "targets/ShareX/ShareX.sln",
121+
"pin": {
122+
"repository": "ShareX/ShareX",
123+
"commit": "0df9ca4d83eed9d2489048c539d7d1fc2860fdec"
124+
},
125+
"alias_of": "oss-ShareX.sln",
126+
"why": "#262's large-solution control. Declared under its own id because the gate names it, and recorded as an ALIAS so it is never counted twice in a denominator"
127+
}
128+
],
129+
"calibration": [
130+
{
131+
"id": "cal-facts-tiny",
132+
"kind": "generated-facts",
133+
"generator": "facts",
134+
"scale": 1,
135+
"why": "instrument-sizing only: the smallest document the door accepts"
136+
},
137+
{
138+
"id": "cal-facts-small",
139+
"kind": "generated-facts",
140+
"generator": "facts",
141+
"scale": 25,
142+
"why": "instrument-sizing only"
143+
},
144+
{
145+
"id": "cal-facts-medium",
146+
"kind": "generated-facts",
147+
"generator": "facts",
148+
"scale": 250,
149+
"why": "instrument-sizing only: large enough that analysis is not lost in startup"
150+
},
151+
{
152+
"id": "cal-facts-refused",
153+
"kind": "generated-facts",
154+
"generator": "refused",
155+
"why": "a document the strict door REFUSES: the parse rung, which stops before bridge and analysis"
156+
},
157+
{
158+
"id": "cal-source-small",
159+
"kind": "generated-source",
160+
"generator": "csharp",
161+
"scale": 4,
162+
"why": "instrument-sizing only: a synthetic C# tree for the launcher rungs"
163+
},
164+
{
165+
"id": "cal-source-medium",
166+
"kind": "generated-source",
167+
"generator": "csharp",
168+
"scale": 40,
169+
"why": "instrument-sizing only"
170+
}
171+
]
172+
}

‎docs/evidence/p022-stage2-census.json‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"C": "COMPARE GATE. Runs both engines under the ratified #260 / Stage-1 differential contract. Stage 2 increases Rust exposure and must never pay for it with differential evidence, so these are neither weakened nor relabelled.",
88
"D": "OPERATIONAL SELF-DOGFOOD. Own.NET running Owen over its own code as the engine under test — not verifying the public default, not acting as a reference. This is the Stage-2 Rust-default population: every Class-D call site selects Rust EXPLICITLY through the ratified launcher contract, with the production `own-cli` supplied via OWEN_RUST_CORE."
99
},
10-
"entry_points_comment": "What counts as executing the core. The launcher surfaces are direct; the wrapper entries are scripts that invoke a launcher or the core themselves, listed so an indirect call site cannot escape the census by adding one level of indirection. Two negative lookaheads are deliberate and were measured, not guessed: `benchmark.py --selftest` validates the scoring and SARIF-parsing logic against embedded fixtures with no SDK, and `shadow_sweep.py --collect`/`--result` assemble and check a record of legs that ran elsewhere. Neither executes the analysis core, so neither is a call site; counting them would inflate the denominator with runs that analyse nothing. Comment lines and `paths:` filters are stripped for the same reason.",
10+
"entry_points_comment": "What counts as executing the core. The launcher surfaces are direct; the wrapper entries are scripts that invoke a launcher or the core themselves, listed so an indirect call site cannot escape the census by adding one level of indirection. That list is a closed enumeration and therefore a maintenance obligation: a NEW wrapper script must be added here when it lands, or the job that runs it is invisible to the census. scripts/perf_baseline.py (#263-A) is the first one added under that rule. Two negative lookaheads are deliberate and were measured, not guessed: `benchmark.py --selftest` validates the scoring and SARIF-parsing logic against embedded fixtures with no SDK, and `shadow_sweep.py --collect`/`--result` assemble and check a record of legs that ran elsewhere. Neither executes the analysis core, so neither is a call site; counting them would inflate the denominator with runs that analyse nothing. Comment lines and `paths:` filters are stripped for the same reason.",
1111
"entry_points": [
1212
"scripts/own-check\\.sh",
1313
"scripts/own-check\\.ps1",
@@ -18,7 +18,8 @@
1818
"scripts/benchmark\\.py(?!.*--selftest)",
1919
"tests/shadow_sweep\\.py(?!.*(--collect|--result))",
2020
"cargo test[^\\n]*-p own-cli",
21-
"scripts/mutate_campaign\\.py"
21+
"scripts/mutate_campaign\\.py",
22+
"scripts/perf_baseline\\.py"
2223
],
2324
"call_sites": [
2425
{
@@ -27,6 +28,12 @@
2728
"class": "B",
2829
"why": "Replays the frozen Python-authored CLI fixture family against the built Rust `own-cli` with zero Python at run time. The measurement IS the parity between the two implementations, so it is engine-specific by construction and has no default to move."
2930
},
31+
{
32+
"workflow": ".github/workflows/ci.yml",
33+
"job": "perf-instrument",
34+
"class": "B",
35+
"why": "#263-A's measurement instrument, calibrating. It drives BOTH engines through the production launcher, but it is not a compare gate: it produces no engine comparison and structurally refuses to, and it measures no decisive workload before the D7 freeze. A deliberately engine-specific measurement whose subject is the instrument rather than either engine."
36+
},
3037
{
3138
"workflow": ".github/workflows/ci.yml",
3239
"job": "stage1-engine",

0 commit comments

Comments
 (0)