Skip to content

Commit 8cd85a0

Browse files
claudeclaude
committed
ci: the shadow sweep — five pinned repositories, large-solution controls, examples, windows path forms (#260 sweep.2)
The scheduled/manual half of #260's CI strategy. The fast PR gate — the committed corpus and the C# samples — is untouched. ONE JOB PER DOCUMENT, not per repository. The directory walk and the `.sln` fan-out are different extractor paths over the same checkout; they fail differently, and a leg that covered both could not say which half a failure belonged to. Nine matrix legs plus an examples job: five repository walks, four solution documents (the largest solution of every target that has one), and `examples/`. MaterialDesignInXamlToolkit has no classic solution at its pinned commit — only the `.slnx` form, which the extractor's resolver does not read — so it has one leg, and the sweep says so rather than averaging it away. THE PIN IS VERIFIED, not trusted. The target is checked out at its sha and `git rev-parse HEAD` is compared with the pin before anything else runs. Drift fails the leg: a target that moved has not been measured, it has been replaced. THE EXTRACTOR RUNS ONCE per document (`own-check.sh --emit-facts` persists the file stage 1 already wrote) and the driver reads that file once, as bytes, through a one-document manifest that carries the document's provenance and its `facts_sha256`. `OWN_SHADOW_ENGINE` is always explicit and the result names the adapter by digest. The reference pack is materialized the way the corpus-benchmark job does it, but NOT `continue-on-error` here: these targets are WPF/WinForms, so a run without the refs would measure a different document than the sweep claims. WINDOWS PATH FORMS are a job, and they promise nothing beyond what runs: the committed-corpus compare on a Windows runner with an adapter built there, plus the driver's own controls under `OWN_SHADOW_COMPARE_REQUIRED=1` — the group that, until this branch, could not execute on Windows at all. THE AGGREGATION exists because every leg can be green and the SWEEP still be a lie: a leg that never ran uploads nothing, and a matrix that lost an entry says nothing at all. It assembles one run record from every leg and checks it against the committed sweep DEFINITION, so the denominator comes from what the sweep declares rather than from whatever happened to be measured. It also refuses if any leg did not succeed. Reproduction artifacts upload on failure only, as #260's CI strategy says; the per-document results and the run summary upload always, because they are the record. Refs #260, #269, #250, #243 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WPozrRezSfnH9RvQGfn1Nn
1 parent 7ecd6a3 commit 8cd85a0

1 file changed

Lines changed: 369 additions & 0 deletions

File tree

‎.github/workflows/shadow-sweep.yml‎

Lines changed: 369 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,369 @@
1+
# The scheduled/manual half of #260's CI strategy: compare mode over the five
2+
# pinned OSS repositories of #243, the large/multi-project solution controls,
3+
# the examples tree, and the Windows path forms. The fast PR gate (the
4+
# committed corpus and the C# samples) lives in ci.yml and is unchanged.
5+
#
6+
# THE RULES THIS WORKFLOW EXISTS TO ENFORCE, none of which a green tick alone
7+
# would show:
8+
#
9+
# * a target is checked out AT ITS PIN and `git rev-parse HEAD` is compared
10+
# with it before anything runs — a target that moved is a failed job, not a
11+
# newer measurement;
12+
# * the extractor runs EXACTLY ONCE per document (`own-check.sh
13+
# --emit-facts` persists the file stage 1 already wrote), and the driver
14+
# reads that file once, as bytes;
15+
# * `OWN_SHADOW_ENGINE` is always set explicitly and the result names the
16+
# adapter by sha256, so a stale build cannot stand in;
17+
# * zero documents compared is a FAILURE, and so is a declared target the run
18+
# never reached — the aggregation re-checks both against the committed
19+
# sweep definition rather than trusting the legs' own exit codes.
20+
#
21+
# Reproduction artifacts upload on failure only (#260's CI strategy). The run
22+
# summary and the per-document results upload always: they are the record.
23+
name: shadow sweep (#260)
24+
25+
on:
26+
workflow_dispatch:
27+
schedule:
28+
# Weekly, Mondays 04:17 UTC. Off the hour on purpose — the top of the hour
29+
# is the busiest slot on shared runners.
30+
- cron: "17 4 * * 1"
31+
32+
permissions:
33+
contents: read
34+
35+
env:
36+
DOCUMENT_OUT: shadow-sweep
37+
38+
jobs:
39+
# One job per DOCUMENT rather than per repository: the directory walk and the
40+
# `.sln` fan-out are different extractor paths over the same checkout, they
41+
# fail differently, and a matrix leg that covered both could not say which
42+
# half a failure belonged to.
43+
document:
44+
name: ${{ matrix.id }}
45+
runs-on: ubuntu-latest
46+
strategy:
47+
fail-fast: false
48+
matrix:
49+
include:
50+
- id: ShareX.repo
51+
target: ShareX
52+
repository: ShareX/ShareX
53+
commit: 0df9ca4d83eed9d2489048c539d7d1fc2860fdec
54+
mode: directory-walk
55+
input: ""
56+
- id: ShareX.sln
57+
target: ShareX
58+
repository: ShareX/ShareX
59+
commit: 0df9ca4d83eed9d2489048c539d7d1fc2860fdec
60+
mode: solution
61+
input: ShareX.sln
62+
- id: MahApps.Metro.repo
63+
target: MahApps.Metro
64+
repository: MahApps/MahApps.Metro
65+
commit: 72099e310bac2d12ac98fd7560b69679252519f5
66+
mode: directory-walk
67+
input: ""
68+
- id: MahApps.Metro.sln
69+
target: MahApps.Metro
70+
repository: MahApps/MahApps.Metro
71+
commit: 72099e310bac2d12ac98fd7560b69679252519f5
72+
mode: solution
73+
input: src/MahApps.Metro.sln
74+
- id: MaterialDesignInXamlToolkit.repo
75+
target: MaterialDesignInXamlToolkit
76+
repository: MaterialDesignInXAML/MaterialDesignInXamlToolkit
77+
commit: ef3a5ea434e39182b1848f5e11aaea6b3890581f
78+
mode: directory-walk
79+
input: ""
80+
- id: AvalonEdit.repo
81+
target: AvalonEdit
82+
repository: icsharpcode/AvalonEdit
83+
commit: ed0bd149059469ac9bd39b13cf8a341b12a6c1da
84+
mode: directory-walk
85+
input: ""
86+
- id: AvalonEdit.sln
87+
target: AvalonEdit
88+
repository: icsharpcode/AvalonEdit
89+
commit: ed0bd149059469ac9bd39b13cf8a341b12a6c1da
90+
mode: solution
91+
input: ICSharpCode.AvalonEdit.sln
92+
- id: ClosedXML.repo
93+
target: ClosedXML
94+
repository: ClosedXML/ClosedXML
95+
commit: 4e89dcedd83cad553e84d2d97f77fc3d7deb630f
96+
mode: directory-walk
97+
input: ""
98+
- id: ClosedXML.sln
99+
target: ClosedXML
100+
repository: ClosedXML/ClosedXML
101+
commit: 4e89dcedd83cad553e84d2d97f77fc3d7deb630f
102+
mode: solution
103+
input: ClosedXML.sln
104+
steps:
105+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
106+
# The target, AT ITS PIN. `persist-credentials: false` because nothing in
107+
# this job pushes anywhere and a third-party checkout has no business
108+
# carrying a token.
109+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
110+
with:
111+
repository: ${{ matrix.repository }}
112+
ref: ${{ matrix.commit }}
113+
path: targets/${{ matrix.target }}
114+
persist-credentials: false
115+
- name: Verify the pin
116+
# Drift is a hard failure of this leg, never a warning: a target that
117+
# moved has not been measured, it has been replaced.
118+
run: |
119+
head=$(git -C "targets/${{ matrix.target }}" rev-parse HEAD)
120+
echo "${{ matrix.target }} HEAD=$head pin=${{ matrix.commit }}"
121+
test "$head" = "${{ matrix.commit }}" || {
122+
echo "PIN DRIFT: ${{ matrix.target }} is at $head, the sweep pins ${{ matrix.commit }}"
123+
exit 1
124+
}
125+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
126+
with:
127+
python-version: "3.13"
128+
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4
129+
with:
130+
dotnet-version: "8.0.x"
131+
- uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10
132+
with:
133+
toolchain: stable
134+
# The WindowsDesktop reference pack, materialized exactly the way the
135+
# corpus-benchmark job does it. NOT continue-on-error here: the pinned
136+
# targets are WPF/WinForms, so a run without the refs measures a
137+
# different document than the sweep says it measured.
138+
- name: Materialize framework reference assemblies
139+
run: |
140+
tmp=$(mktemp -d)
141+
printf '%s\n' \
142+
'<Project Sdk="Microsoft.NET.Sdk">' \
143+
' <PropertyGroup>' \
144+
' <TargetFramework>net8.0-windows</TargetFramework>' \
145+
' <UseWPF>true</UseWPF>' \
146+
' <UseWindowsForms>true</UseWindowsForms>' \
147+
' <EnableWindowsTargeting>true</EnableWindowsTargeting>' \
148+
' </PropertyGroup>' \
149+
'</Project>' > "$tmp/ref.csproj"
150+
dotnet restore "$tmp/ref.csproj" >/dev/null
151+
d=$(find "$HOME/.nuget/packages/microsoft.windowsdesktop.app.ref" -type d -name 'net8.0' | sort | tail -1)
152+
test -n "$d" || { echo "the WindowsDesktop reference pack did not resolve"; exit 1; }
153+
echo "OWN_EXTRA_REF_DIRS=$d" >> "$GITHUB_ENV"
154+
echo "framework refs: $d ($(find "$d" -name '*.dll' | wc -l) dlls)"
155+
- name: Build the dev-only engine adapter
156+
working-directory: rust
157+
run: cargo build --release -p own-shadow --bin own-shadow-engine
158+
# ONE extraction. `--emit-facts` persists the file stage 1 already wrote;
159+
# stage 2 is the reference's verdict path and is neither a second
160+
# extraction nor the comparison.
161+
- name: Extract the OwnIR facts, exactly once
162+
run: |
163+
input="targets/${{ matrix.target }}"
164+
if [ -n "${{ matrix.input }}" ]; then
165+
input="targets/${{ matrix.target }}/${{ matrix.input }}"
166+
fi
167+
echo "extracting: $input"
168+
scripts/own-check.sh --format sarif --severity warning \
169+
--emit-facts "$RUNNER_TEMP/facts.json" -- "$input" \
170+
> "$RUNNER_TEMP/findings.sarif.json"
171+
ls -l "$RUNNER_TEMP/facts.json"
172+
- name: Describe the document
173+
run: |
174+
python - <<'PY'
175+
import hashlib, json, os
176+
facts = os.path.join(os.environ["RUNNER_TEMP"], "facts.json")
177+
with open(facts, "rb") as f:
178+
digest = hashlib.sha256(f.read()).hexdigest()
179+
target = "${{ matrix.target }}"
180+
rel = "${{ matrix.input }}"
181+
where = f"targets/{target}" + (f"/{rel}" if rel else "")
182+
manifest = {
183+
"schema": 1,
184+
"targets": [target],
185+
"documents": [{
186+
"source": facts,
187+
"target": target,
188+
"target_commit": "${{ matrix.commit }}",
189+
"extraction_mode": "${{ matrix.mode }}",
190+
"extraction_command": (
191+
"OWN_EXTRA_REF_DIRS=<WindowsDesktop ref pack net8.0> "
192+
"scripts/own-check.sh --format sarif --severity warning "
193+
f"--emit-facts <FACTS> -- {where}"),
194+
"facts_sha256": digest,
195+
"timeout_seconds": 600.0,
196+
}],
197+
}
198+
with open(os.path.join(os.environ["RUNNER_TEMP"], "manifest.json"), "w") as f:
199+
json.dump(manifest, f, indent=2)
200+
print(json.dumps(manifest, indent=2))
201+
PY
202+
- name: Compare mode over the document
203+
env:
204+
OWN_SHADOW_ENGINE: ${{ github.workspace }}/rust/target/release/own-shadow-engine
205+
run: |
206+
python scripts/shadow_compare.py --engine compare \
207+
--manifest "$RUNNER_TEMP/manifest.json" \
208+
--out "$RUNNER_TEMP/$DOCUMENT_OUT"
209+
# The record uploads whether or not the run agreed; a leg that only
210+
# uploaded on failure would leave the aggregation unable to tell a green
211+
# target from a skipped one.
212+
- name: Upload the run record
213+
if: always()
214+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
215+
with:
216+
name: shadow-sweep-${{ matrix.id }}
217+
path: ${{ runner.temp }}/${{ env.DOCUMENT_OUT }}
218+
retention-days: 30
219+
if-no-files-found: warn
220+
221+
# The examples tree: cheap, named in #260's test matrix, and the one document
222+
# of this sweep that needs no third-party checkout.
223+
examples:
224+
name: examples
225+
runs-on: ubuntu-latest
226+
steps:
227+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
228+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
229+
with:
230+
python-version: "3.13"
231+
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4
232+
with:
233+
dotnet-version: "8.0.x"
234+
- uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10
235+
with:
236+
toolchain: stable
237+
- name: Build the dev-only engine adapter
238+
working-directory: rust
239+
run: cargo build --release -p own-shadow --bin own-shadow-engine
240+
- name: Extract the OwnIR facts, exactly once
241+
run: |
242+
scripts/own-check.sh --format sarif --severity warning \
243+
--emit-facts "$RUNNER_TEMP/facts.json" -- examples \
244+
> "$RUNNER_TEMP/findings.sarif.json"
245+
- name: Describe the document
246+
run: |
247+
python - <<'PY'
248+
import hashlib, json, os
249+
facts = os.path.join(os.environ["RUNNER_TEMP"], "facts.json")
250+
with open(facts, "rb") as f:
251+
digest = hashlib.sha256(f.read()).hexdigest()
252+
manifest = {
253+
"schema": 1,
254+
"targets": ["examples"],
255+
"documents": [{
256+
"source": facts,
257+
"target": "examples",
258+
"target_commit": os.environ["GITHUB_SHA"],
259+
"extraction_mode": "directory-walk",
260+
"extraction_command": (
261+
"scripts/own-check.sh --format sarif --severity warning "
262+
"--emit-facts <FACTS> -- examples"),
263+
"facts_sha256": digest,
264+
"timeout_seconds": 600.0,
265+
}],
266+
}
267+
with open(os.path.join(os.environ["RUNNER_TEMP"], "manifest.json"), "w") as f:
268+
json.dump(manifest, f, indent=2)
269+
PY
270+
- name: Compare mode over the examples document
271+
env:
272+
OWN_SHADOW_ENGINE: ${{ github.workspace }}/rust/target/release/own-shadow-engine
273+
run: |
274+
python scripts/shadow_compare.py --engine compare \
275+
--manifest "$RUNNER_TEMP/manifest.json" \
276+
--out "$RUNNER_TEMP/$DOCUMENT_OUT"
277+
- name: Upload the run record
278+
if: always()
279+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
280+
with:
281+
name: shadow-sweep-examples
282+
path: ${{ runner.temp }}/${{ env.DOCUMENT_OUT }}
283+
retention-days: 30
284+
if-no-files-found: warn
285+
286+
# Windows path forms, MEASURED and nothing promised beyond what runs: the
287+
# committed-corpus gate, on a Windows runner, with an adapter built there.
288+
# The driver's own controls run too and cannot skip — that group is the one
289+
# that had never executed on Windows at all.
290+
windows-path-forms:
291+
name: windows path forms (committed corpus)
292+
runs-on: windows-latest
293+
defaults:
294+
run:
295+
shell: bash
296+
steps:
297+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
298+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
299+
with:
300+
python-version: "3.13"
301+
- uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10
302+
with:
303+
toolchain: stable
304+
- name: Build the dev-only engine adapter
305+
working-directory: rust
306+
run: cargo build --release -p own-shadow --bin own-shadow-engine
307+
- name: Compare mode over the committed corpus
308+
env:
309+
OWN_SHADOW_ENGINE: ${{ github.workspace }}/rust/target/release/own-shadow-engine.exe
310+
run: python scripts/shadow_compare.py --engine compare --corpus --quiet --out "$RUNNER_TEMP/shadow"
311+
- name: The compare driver's controls (adapter required)
312+
env:
313+
OWN_SHADOW_ENGINE: ${{ github.workspace }}/rust/target/release/own-shadow-engine.exe
314+
OWN_SHADOW_COMPARE_REQUIRED: "1"
315+
run: python tests/test_shadow_compare.py
316+
- name: Upload the divergence reports
317+
if: failure()
318+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
319+
with:
320+
name: shadow-sweep-windows-reports
321+
path: ${{ runner.temp }}/shadow
322+
retention-days: 14
323+
if-no-files-found: ignore
324+
325+
# The aggregation, and the reason it exists: every leg above can be green and
326+
# the SWEEP still be a lie, because a leg that never ran uploads nothing and a
327+
# matrix that lost an entry says nothing at all. This job assembles one run
328+
# record from every leg and checks it against the committed sweep DEFINITION —
329+
# the denominator lives there, not in whatever happened to be measured.
330+
aggregate:
331+
name: aggregate the sweep
332+
runs-on: ubuntu-latest
333+
needs: [document, examples, windows-path-forms]
334+
if: always()
335+
steps:
336+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
337+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
338+
with:
339+
python-version: "3.13"
340+
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
341+
with:
342+
pattern: shadow-sweep-*
343+
path: ${{ runner.temp }}/legs
344+
- name: Every leg must have run
345+
run: |
346+
echo "document: ${{ needs.document.result }}"
347+
echo "examples: ${{ needs.examples.result }}"
348+
echo "windows path forms: ${{ needs['windows-path-forms'].result }}"
349+
fail=0
350+
for r in "${{ needs.document.result }}" "${{ needs.examples.result }}" \
351+
"${{ needs['windows-path-forms'].result }}"; do
352+
[ "$r" = "success" ] || fail=1
353+
done
354+
test "$fail" -eq 0 || { echo "a leg of the sweep did not succeed"; exit 1; }
355+
- name: Assemble one run record
356+
run: |
357+
python tests/shadow_sweep.py --collect "$RUNNER_TEMP/legs" \
358+
--write "$RUNNER_TEMP/p022-shadow-sweep.result.json" \
359+
--workflow-run-url "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
360+
- name: Check it against the committed sweep definition
361+
run: python tests/shadow_sweep.py --result "$RUNNER_TEMP/p022-shadow-sweep.result.json"
362+
- name: Upload the run summary
363+
if: always()
364+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
365+
with:
366+
name: shadow-sweep-run-summary
367+
path: ${{ runner.temp }}/p022-shadow-sweep.result.json
368+
retention-days: 90
369+
if-no-files-found: warn

0 commit comments

Comments
 (0)