Skip to content

Commit 18ea4a9

Browse files
claudeclaude
committed
docs(p022): inventory the #260 sweep before running it (#260 sweep.0)
The inventory the measurement is taken against, written before anything ran, so that the matrix this work promises is one somebody measured rather than one somebody estimated. THE PINS are #243's, reused verbatim and copied from the machine-readable remeasure data rather than from the prose beside it. The rule is stated where it can be enforced: a checkout materializes the COMMIT (a depth-1 fetch of the sha, detached), and `git rev-parse HEAD` is compared with the pin before anything runs. A target that moved is a failed target, not a newer measurement. WHAT THIS ENVIRONMENT CAN DO, measured rather than assumed: a .NET 10 SDK that builds the extractor's pinned net8.0 target, the 8.0.30 runtimes it needs, Roslyn pinned by the extractor itself so the SDK version does not move the parse, and a Rust toolchain installed for this work beside the MSVC build tools that were already here. The whole matrix runs here, so the fallback of taking the measurement through the workflow and reading its artifacts back is not used. The host is Windows, which is why the path-form leg is a local measurement here AND a windows-latest job in CI. WHICH TARGETS CARRY A SOLUTION, measured from the checkout by resolving each `.sln`'s member projects on disk. Four of the five do; MaterialDesignInXaml- Toolkit carries only the XML solution format at its pinned commit, which the extractor's classic-format resolver does not read. That is recorded as a measured gap and acted on nowhere: teaching the extractor `.slnx` would be a production change, and this task changes no production behaviour. The target is covered by its directory walk alone and the record says so per target rather than averaging it away. THE SOLUTION DOCUMENTS ARE NOT THE WALK DOCUMENTS UNDER ANOTHER NAME, and that is measured too: on ShareX the two carry the same components and functions in a different ORDER — the fan-out enumerates project by project, the walk in directory order — so they have equal byte length and different digests. Two documents, a different extractor path, and per-layer ordering semantics that are declared rather than normalized away. WHAT "COVERED" MEANS, written down before there is anything to be tempted by: a repository is not covered because extraction succeeded, and a solution is not covered because some project inside it emitted OwnIR. Coverage is a recorded, non-empty set of documents fed byte-identically to both engines and judged by compare mode, per target, with the denominator on the record — the fifth failure mode of #250, restated as three rules the driver will enforce rather than three sentences a note asserts. The churn budget, the ledger schema and the freeze on the comparison machinery are in §2.6 and §2.5. No count this sweep produces appears here; the inventory figures are inputs, and every outcome count will be a link into docs/generated/. Refs #260, #269, #250, #243 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPozrRezSfnH9RvQGfn1Nn
1 parent b05b38a commit 18ea4a9

1 file changed

Lines changed: 283 additions & 0 deletions

File tree

‎docs/notes/p022-shadow-sweep.md‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
# P-022 step 7a (#260) — the sweep: five pinned repositories, large-solution controls, examples, path forms
2+
3+
> **Scope of this note.** It is the record of #260's **final acceptance
4+
> measurement** — the part of its test matrix the acceptance surfaces over the
5+
> committed corpus deliberately did not take: the five pinned OSS repositories
6+
> of #243 at their pinned commits, the large/multi-project solution controls,
7+
> the `examples/` tree, and the Windows path forms. The contract it runs under
8+
> is D-4..D-7, B-2, B-3, R-1 and R-2 in the
9+
> [owner-decision ledger](p022-shadow-infra-owner-decisions.md); this note may
10+
> not reopen any of them. What the committed corpus already proves is
11+
> [the acceptance note](p022-shadow-acceptance.md); this is its §6 first two
12+
> bullets being taken rather than restated.
13+
14+
**Where the numbers are.** Every count this sweep *produced* — documents,
15+
outcomes, denominators per target, byte sizes, wall-clock — lives in
16+
[`docs/generated/p022-shadow-sweep.md`](../generated/p022-shadow-sweep.md),
17+
rendered by `scripts/render_checkpoint_status.py` from the recorded run, and is
18+
reached from here by link. The inventory below is the other kind of number: the
19+
*inputs* measured from the checkouts before anything ran, so that the matrix
20+
this note promises is a matrix somebody measured rather than one somebody
21+
estimated. Those are stated here and also carried as data in
22+
[`docs/evidence/p022-shadow-sweep.json`](../evidence/p022-shadow-sweep.json).
23+
24+
---
25+
26+
## §1 — The targets, the pins, and what this environment can actually do
27+
28+
### 1.1 The five, and the pin rule
29+
30+
The targets and commits are #243's, **reused verbatim** and never re-resolved
31+
against upstream HEAD. They are copied from
32+
[`docs/notes/precision-remeasure-2026-07-12-data/`](precision-remeasure-2026-07-12-data/),
33+
which is the machine-readable record of the last sweep that used them, not from
34+
the prose of the note beside it.
35+
36+
| target | upstream | pinned commit |
37+
|---|---|---|
38+
| ShareX | `ShareX/ShareX` | `0df9ca4d83eed9d2489048c539d7d1fc2860fdec` |
39+
| MahApps.Metro | `MahApps/MahApps.Metro` | `72099e310bac2d12ac98fd7560b69679252519f5` |
40+
| MaterialDesignInXamlToolkit | `MaterialDesignInXAML/MaterialDesignInXamlToolkit` | `ef3a5ea434e39182b1848f5e11aaea6b3890581f` |
41+
| AvalonEdit | `icsharpcode/AvalonEdit` | `ed0bd149059469ac9bd39b13cf8a341b12a6c1da` |
42+
| ClosedXML | `ClosedXML/ClosedXML` | `4e89dcedd83cad553e84d2d97f77fc3d7deb630f` |
43+
44+
**The pin rule.** A checkout is materialized by fetching *the commit itself*
45+
(`git fetch --depth 1 origin <sha>`, then `git checkout --detach FETCH_HEAD`),
46+
so what lands is the pin rather than a branch tip that happens to contain it.
47+
Before anything else runs, `git rev-parse HEAD` is compared with the pin. **A
48+
mismatch is a hard failure of that target, not a warning and not a newer
49+
measurement.** A target that moved has not been measured; it has been replaced.
50+
51+
The same rule applies to the *port's* half. The adapter is named in every
52+
recorded comparison by `sha256` and byte length as well as by path, and
53+
`OWN_SHADOW_ENGINE` is always set explicitly — a driver that found
54+
`rust/target/debug/own-shadow-engine` because the environment was unset
55+
compared *an* engine, not *the* engine.
56+
57+
### 1.2 What this environment is
58+
59+
This environment **can** run the whole matrix, so it does: the G.0 fallback
60+
(take the measurement through the workflow and read its artifacts back) is not
61+
used, and the recorded run is a local one whose commands, commits and artifact
62+
identities are on the record. What made that possible, measured rather than
63+
assumed:
64+
65+
| capability | what is present |
66+
|---|---|
67+
| .NET SDK | 10.0.400 — builds the extractor's pinned `net8.0` target |
68+
| .NET runtimes | `Microsoft.NETCore.App` 8.0.30 and `Microsoft.WindowsDesktop.App` 8.0.30 |
69+
| Roslyn | pinned by the extractor itself (`Microsoft.CodeAnalysis.CSharp` 4.9.2), so the SDK version does not move the parse |
70+
| Rust | 1.98.1 (`x86_64-pc-windows-msvc`), installed for this work; MSVC 14.44 + Windows SDK 10.0.26100 were already present |
71+
| host | Windows 11 — which is why the path-form leg of the matrix is a *local* measurement here and a `windows-latest` job in CI |
72+
73+
The reference pack is materialized exactly the way `ci.yml`'s
74+
`corpus-benchmark` job does it — a scratch `net8.0-windows` project with
75+
`UseWPF` / `UseWindowsForms` / `EnableWindowsTargeting`, `dotnet restore`, then
76+
the resolved `ref/net8.0` directory exported as `OWN_EXTRA_REF_DIRS`. It
77+
resolves to `microsoft.windowsdesktop.app.ref` **8.0.30** here; the 2026-07-12
78+
remeasure resolved **8.0.28**. Both carry 47 DLLs. That difference is recorded
79+
rather than pinned away because **it cannot bias this measurement**: whatever it
80+
does to a facts document, both engines receive that document as the same
81+
bytes. It would matter to a precision remeasure; this is not one.
82+
83+
### 1.3 The disk and wall-clock the matrix costs
84+
85+
Measured on this machine before the matrix was promised, one target extracted
86+
first (AvalonEdit, the smallest) and then the rest:
87+
88+
| item | measured |
89+
|---|---|
90+
| the five checkouts, on disk | ~147 MB total (AvalonEdit ~3 MB, ClosedXML ~26 MB, MahApps.Metro ~32 MB, ShareX ~41 MB, MaterialDesign ~45 MB) |
91+
| extraction, all ten documents | ~98 s total; the slowest single document is ShareX's directory walk at ~20 s |
92+
| the ten facts documents | ~1.1 MB total; the largest is ~250 KB |
93+
| compare, per document | well under a second — the committed-corpus gate runs 104 documents in ~0.7 s |
94+
95+
The consequence for the timeout policy is in §2.4: nothing here is near the
96+
driver's default, and the default is therefore kept and *recorded* rather than
97+
raised on a guess.
98+
99+
---
100+
101+
## §2 — The matrix, exactly as it will be run
102+
103+
### 2.1 Which targets carry a solution — measured, from the checkout
104+
105+
Counted by resolving each `.sln`'s `Project(...)` entries to `.csproj` members
106+
that exist on disk, and counting the `.cs` files under those members:
107+
108+
| target | solutions found | member projects | `.cs` under members |
109+
|---|---|---|---|
110+
| ShareX | `ShareX.sln` | 13 | 1126 |
111+
| | `ShareX.ImageEditor.sln` | 2 | 433 |
112+
| ClosedXML | `ClosedXML.sln` | 6 | 921 |
113+
| MahApps.Metro | `src/MahApps.Metro.sln` | 4 | 328 |
114+
| AvalonEdit | `ICSharpCode.AvalonEdit.sln` | 3 | 248 |
115+
| | `ICSharpCode.AvalonEdit.Documentation.sln` | 1 | 214 |
116+
| MaterialDesignInXamlToolkit | **none** — `MaterialDesignToolkit.Full.slnx` only | — | — |
117+
118+
**MaterialDesignInXamlToolkit has no classic solution at its pinned commit.**
119+
It carries the XML solution format (`.slnx`), and the extractor's solution
120+
resolver reads the classic `Project("{...}") = "name", "path"` form only. That
121+
is a *measured gap*, recorded here and nowhere else acted on: teaching the
122+
extractor `.slnx` would be a production change, and this task changes no
123+
production behaviour. The target is therefore covered by its directory walk
124+
alone, and the record says so per target rather than averaging it away.
125+
126+
### 2.2 The large-solution controls taken
127+
128+
#260 asks for "selected large/multi-project solutions" and this task's brief for
129+
"at least the two largest". Taken: **the largest solution of every target that
130+
has one** — `ShareX.sln` (13 members), `ClosedXML.sln` (6), `MahApps.Metro.sln`
131+
(4), `ICSharpCode.AvalonEdit.sln` (3). The two largest are the first two; the
132+
other two are nearly free and make the mode's coverage per-target rather than
133+
per-sample. `ShareX.ImageEditor.sln` and the AvalonEdit documentation solution
134+
are proper subsets of the solutions already taken and are skipped for that
135+
reason, which is a choice on the record rather than a silent omission.
136+
137+
The solution documents are **not** the directory-walk documents under another
138+
name. Measured on ShareX: the two documents carry the same 106 components and
139+
74 functions, in a **different order** — the fan-out enumerates the source set
140+
project by project, the walk enumerates it in directory order — so they have
141+
identical byte length and different digests, and they are two documents, not
142+
one. That is exactly the property a large-solution control is for: a different
143+
extractor path, a differently ordered document, and per-layer ordering
144+
semantics that are *declared* rather than normalized away.
145+
146+
### 2.3 The ten documents, and the commands, verbatim
147+
148+
Ten documents, each extracted **exactly once**. `scripts/own-check.sh` runs the
149+
extractor once into a temporary file and `--emit-facts <path>` persists exactly
150+
that file; stage 2 (`python -m ownlang ownir`) is the reference's *verdict* path
151+
and is neither a second extraction nor the comparison. The comparison is the
152+
driver's, taken from the emitted bytes.
153+
154+
```bash
155+
# per document, with <REFDIR> the materialized WindowsDesktop ref pack (§1.2)
156+
OWN_EXTRA_REF_DIRS=<REFDIR> scripts/own-check.sh \
157+
--format sarif --severity warning \
158+
--emit-facts <FACTS> -- <INPUT>
159+
```
160+
161+
| document | mode | `<INPUT>` |
162+
|---|---|---|
163+
| `ShareX.repo` | directory walk | `<checkout>/ShareX` |
164+
| `MahApps.Metro.repo` | directory walk | `<checkout>/MahApps.Metro` |
165+
| `MaterialDesignInXamlToolkit.repo` | directory walk | `<checkout>/MaterialDesignInXamlToolkit` |
166+
| `AvalonEdit.repo` | directory walk | `<checkout>/AvalonEdit` |
167+
| `ClosedXML.repo` | directory walk | `<checkout>/ClosedXML` |
168+
| `ShareX.sln` | `.sln` fan-out | `<checkout>/ShareX/ShareX.sln` |
169+
| `ClosedXML.sln` | `.sln` fan-out | `<checkout>/ClosedXML/ClosedXML.sln` |
170+
| `MahApps.Metro.sln` | `.sln` fan-out | `<checkout>/MahApps.Metro/src/MahApps.Metro.sln` |
171+
| `AvalonEdit.sln` | `.sln` fan-out | `<checkout>/AvalonEdit/ICSharpCode.AvalonEdit.sln` |
172+
| `examples` | directory walk | `<Own.NET>/examples` |
173+
174+
The driver then runs **once** over a manifest naming all ten:
175+
176+
```bash
177+
cd rust && cargo build --release -p own-shadow --bin own-shadow-engine
178+
OWN_SHADOW_ENGINE=<the adapter just built> \
179+
python scripts/shadow_compare.py --engine compare \
180+
--manifest <manifest.json> --out <run dir> --quiet
181+
```
182+
183+
and the Windows path-form leg is the committed-corpus gate, on this host, with
184+
the adapter built on this host:
185+
186+
```bash
187+
OWN_SHADOW_ENGINE=<the adapter just built> \
188+
python scripts/shadow_compare.py --engine compare --corpus --quiet
189+
OWN_SHADOW_ENGINE=<the adapter just built> OWN_SHADOW_COMPARE_REQUIRED=1 \
190+
python tests/test_shadow_compare.py
191+
```
192+
193+
### 2.4 What "covered" means here, and the timeout policy
194+
195+
**A repository is not covered because extraction succeeded, and a solution is
196+
not covered because some project inside it emitted OwnIR.** Coverage is a
197+
recorded, non-empty set of OwnIR documents fed byte-identically to both engines
198+
and judged by compare mode — *per target, with the denominator on the record*.
199+
This is the direct descendant of the fifth failure mode in #250: a green gate
200+
over an empty set is worse than a red one, because a red one at least says it
201+
is awake. Hence three rules the driver enforces rather than this note asserting
202+
them:
203+
204+
* a run that compared **zero** documents is a **failure**, never agreement;
205+
* a **target** whose compare-attempted count is zero is a **failed target**,
206+
never a passed repository;
207+
* every document's `facts_sha256` in the manifest is checked against the bytes
208+
the driver actually read, and every document is validated **before any engine
209+
runs**.
210+
211+
**Timeouts.** Explicit per document and recorded per document. §1.3 measured the
212+
whole matrix as sub-second per compare, three orders of magnitude inside the
213+
driver's 120 s default, so the default is what the manifest carries — stated as
214+
a number in the manifest rather than inherited silently, so that raising it for
215+
a future target is a visible edit. A timeout is an **execution failure with a
216+
report** (R-2), never a silent skip.
217+
218+
### 2.5 The sweep ledger schema
219+
220+
Two documents, the same split every campaign in this tree uses — a
221+
**definition** that says what should be measured and a **result** that records
222+
one actual run of it, with one interpreter reading both.
223+
224+
`docs/evidence/p022-shadow-sweep.json` — the definition:
225+
226+
```text
227+
schema, sweep, description
228+
adapter_build_command how the port's half is produced
229+
driver_version the shadow_compare_version this definition expects
230+
timeout_seconds the default the manifest carries
231+
targets[] { target, upstream, pinned_commit, solutions[], slnx[], notes }
232+
documents[] { id, target, extraction_mode, extraction_command, input,
233+
timeout_seconds }
234+
```
235+
236+
`docs/evidence/p022-shadow-sweep.result.json` — one run:
237+
238+
```text
239+
schema, sweep, definition, definition_sha256
240+
source_commit, recorded_at, host, workflow_run_url (null for a local run)
241+
adapter { path, sha256, bytes }
242+
driver_version
243+
documents[] { id, source, target, target_commit, extraction_mode,
244+
extraction_command, facts_sha256, raw { digest, bytes },
245+
canonical { digest, bytes }, outcome, timeout_seconds,
246+
reduction_outcome, by_kind{}, by_acceptance{},
247+
derived_outcome, wall_clock_seconds }
248+
targets[] { target, documents_extracted, compare_attempted, agreed,
249+
diverged, execution_failures, input_refusals,
250+
input_disagreements, declared_boundary_observations,
251+
acceptance_unexplained_observations }
252+
totals { the same fields, summed }
253+
```
254+
255+
`tests/shadow_sweep.py` is the single interpreter of the pair.
256+
`scripts/render_checkpoint_status.py` renders
257+
`docs/generated/p022-shadow-sweep.md` from it and
258+
`tests/test_checkpoint_status.py` gates that the committed fragment equals the
259+
projection — the same pipeline every campaign count in this repository already
260+
goes through. **A re-run replaces the result whole; it is never patched.**
261+
262+
### 2.6 The churn budget, written before anything moved
263+
264+
* `tests/fixtures/` — **nothing**, unless a divergence produces an artifact
265+
small enough to be a control, in which case it is added insertion-stable and
266+
the finding that produced it is on the record with it. The frozen goldens do
267+
not move for a measurement.
268+
* `docs/evidence/` — gains the sweep definition and one recorded result, plus
269+
one campaign definition and result over the driver's new pieces.
270+
* `docs/generated/` — gains `p022-shadow-sweep.md` and the campaign fragment
271+
regenerates.
272+
* **The five repositories' facts documents do not enter the repository.** They
273+
are large and exactly reproducible from the pins and the commands in §2.3;
274+
their *identities* (the raw digest and byte length of each, and the canonical
275+
identity both engines derived) are what the result records.
276+
* **Production behaviour changes nowhere.** Not in `ownlang/`, not in the
277+
analyses, not in the renderers, not in `own-check.sh`'s verdict path. The
278+
driver, the adapter, the workflow and the evidence pipeline are dev tooling.
279+
* **The comparison machinery is frozen for the measurement.** The canonical
280+
form, the artifact verifier, the trace, the reducer, `BOUNDARY_POLICY`, the
281+
derived-SARIF configuration and the driver's *judgement* do not move while
282+
the sweep runs. A divergence is a finding first; adjusting a normalizer to
283+
make one go away is a contract change that waits for the owner.

0 commit comments

Comments
 (0)