From 2f6b6195e4bc8d7c6ae1fbe8209a3f2ee0b25584 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 01:27:14 +0000 Subject: [PATCH 01/22] feat(launcher): opt-in Rust engine behind the existing launcher (#262 Stage 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Python remains the default and the reference on every surface. Rust becomes explicitly selectable, and a compare mode runs both engines over one captured input. Nothing public defaults to Rust and Python distribution is untouched. D1 — one explicit launcher selector, `--engine python|rust|compare`, on all four production launcher surfaces: `owen check`, scripts/own-check.sh, scripts/own-check.ps1 and action.yml (which exposes it as an `engine` input and delegates the semantics to own-check.sh rather than re-implementing them). `own-cli` gets no selector: it presents one engine and knows nothing of Python. D3/D3.1 — Rust/compare resolve the candidate binary from OWEN_RUST_CORE and from nothing else: no PATH discovery, no rust/target probing, no "first binary found", because discovery is how a stale binary silently stands in for the one under test. A missing, empty, nonexistent, non-file or non-executable locator, detected before the selected core has started, is a configuration error — public exit 2, one actionable diagnostic, and never a fallback to Python. The ordering finding this stage repairs: the launcher used to resolve Python and unpack the vendored core unconditionally, because there was only ever one engine. Engine selection now happens before any engine-specific resolution, and a Rust-only run resolves no interpreter and unpacks no Python core — measured by running it with OWEN_PYTHON pointed at a path that is not a Python. D5 — an unexpected Rust child status takes Owen's public internal-error path (5) with the raw status retained in the diagnostic report's typed, nullable `child_exit_code`; the report schema is bumped to 2. 70 is the engines' shared internal-error code: known, still a failure, never read as findings or clean. D4/D4.1 — compare extracts once, captures the OwnIR bytes once, materialises each engine's input from that single value and re-hashes both against it before either engine starts. Streams are compared as raw bytes, not decoded text: a claim about bytes cannot be measured on strings. Agreement exposes the reference's result; divergence and execution failure both exit 5 with an actionable diagnostic and reproduction evidence, and neither ever substitutes one engine's answer for the other's. A zero-document compare fails rather than reporting agreement over nothing. Controls: tests/test_stage1_engine.py drives real launchers against a real candidate and covers the fifteen ratified adversarial controls, without fail-fast, forcing failures through #261's own fault-injection feature rather than through a mock. OWEN_STAGE1_REQUIRE=1 turns a skipped control into a failure so the denominator cannot quietly shrink. docs/evidence/p022-stage1-1 is the mutation campaign over these surfaces. The `--config` carrier stays Python under every engine: it is a separately documented non-core Python duty and #262's D9 tail, still an open owner decision, so neither skipping nor reimplementing it here would be Stage-1 work. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- action.yml | 23 +- docs/evidence/p022-stage1-1.json | 203 +++++ frontend/roslyn/OwnSharp.Cli/CheckCommand.cs | 169 +++-- frontend/roslyn/OwnSharp.Cli/CompareMode.cs | 339 +++++++++ frontend/roslyn/OwnSharp.Cli/CrashReport.cs | 29 +- frontend/roslyn/OwnSharp.Cli/EngineRunner.cs | 139 ++++ .../roslyn/OwnSharp.Cli/EngineSelection.cs | 106 +++ frontend/roslyn/OwnSharp.Cli/Program.cs | 11 + .../roslyn/OwnSharp.Cli/RustCoreLocator.cs | 146 ++++ scripts/own-check.ps1 | 168 ++++- scripts/own-check.sh | 189 ++++- tests/stage1_campaign_layer.sh | 49 ++ tests/test_stage1_engine.py | 705 ++++++++++++++++++ 13 files changed, 2207 insertions(+), 69 deletions(-) create mode 100644 docs/evidence/p022-stage1-1.json create mode 100644 frontend/roslyn/OwnSharp.Cli/CompareMode.cs create mode 100644 frontend/roslyn/OwnSharp.Cli/EngineRunner.cs create mode 100644 frontend/roslyn/OwnSharp.Cli/EngineSelection.cs create mode 100644 frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs create mode 100755 tests/stage1_campaign_layer.sh create mode 100644 tests/test_stage1_engine.py diff --git a/action.yml b/action.yml index 10623b3c..fa8d0779 100644 --- a/action.yml +++ b/action.yml @@ -32,6 +32,18 @@ inputs: description: "How findings are shown: error (default) or warning (advisory)." required: false default: "error" + engine: + description: >- + Which analysis engine runs (#262 Stage 1): python (the DEFAULT and the + reference), rust (the Rust core `own-cli ownir`), or compare (both over + one captured input, exposing the reference's result only when they agree + byte for byte). rust and compare require the candidate binary's absolute + path in the OWEN_RUST_CORE environment variable — the Action does no + discovery, so an unset or unusable OWEN_RUST_CORE is a configuration + error (exit 2) and never a silent fall back to Python. compare is a + development/CI seam for the migration, not yet a promised feature. + required: false + default: "python" fail-on-finding: description: >- Whether a FINDING fails the step. Default false: findings are published @@ -89,10 +101,15 @@ runs: OWN_PATH: ${{ inputs.path }} OWN_FORMAT: ${{ inputs.format }} OWN_SEVERITY: ${{ inputs.severity }} + OWN_ENGINE: ${{ inputs.engine }} OWN_FAIL_ON_FINDING: ${{ inputs.fail-on-finding }} OWN_SARIF_FILE: ${{ inputs.sarif-file }} OWN_CONFIG: ${{ inputs.config }} run: | + # D2: the Action is one of the four launcher surfaces, but its + # engine-selection SEMANTICS are own-check.sh's — it delegates rather + # than re-implementing them, so the real fan-out is smaller than four + # and there is exactly one contract to keep true. check="${{ github.action_path }}/scripts/own-check.sh" # P-035: forward an explicit own.toml to own-check when the caller set one. # Passed as data via OWN_CONFIG (never interpolated into the script body). @@ -112,7 +129,8 @@ runs: sarif="${OWN_SARIF_FILE:-$RUNNER_TEMP/owen.sarif}" set +e "$check" --root "${{ github.action_path }}" --format sarif \ - --severity "$OWN_SEVERITY" "${config_args[@]}" --fail-on-finding -- "$OWN_PATH" > "$sarif" + --severity "$OWN_SEVERITY" --engine "$OWN_ENGINE" "${config_args[@]}" \ + --fail-on-finding -- "$OWN_PATH" > "$sarif" rc=$? set -e echo "sarif-file=$sarif" >> "$GITHUB_OUTPUT" @@ -143,7 +161,8 @@ runs: # action is allowed to negotiate about. set +e "$check" --root "${{ github.action_path }}" --format "$OWN_FORMAT" \ - --severity "$OWN_SEVERITY" "${config_args[@]}" --fail-on-finding -- "$OWN_PATH" + --severity "$OWN_SEVERITY" --engine "$OWN_ENGINE" "${config_args[@]}" \ + --fail-on-finding -- "$OWN_PATH" rc=$? set -e if [ "$rc" -ge 2 ]; then diff --git a/docs/evidence/p022-stage1-1.json b/docs/evidence/p022-stage1-1.json new file mode 100644 index 00000000..6112fc03 --- /dev/null +++ b/docs/evidence/p022-stage1-1.json @@ -0,0 +1,203 @@ +{ + "schema": 1, + "comment": "GENERATED-BY-HAND definition; the RESULT beside it is recorded by scripts/mutate_campaign.py --run and the counts are derived from it, never typed.", + "campaign": "p022-stage1-1", + "description": "#262 Stage 1 — the launcher's engine-selection contract: that Python stays the default, that an explicitly selected Rust core actually runs (and needs no Python), that a Rust failure is never a Python success, that an unexpected child status becomes public exit 5 with the raw status retained, that an unusable OWEN_RUST_CORE is a configuration error rather than a fallback, and that compare extracts once, feeds both engines the same bytes, and refuses to answer when they disagree or when either fails. Every mutation is a plausible MISREADING of that contract rather than a syntactic accident: each one would pass a reviewer who had read the stage's summary instead of its rulings. Every mutation edits a PRODUCTION launcher surface, and every declared layer runs for every mutation (no fail-fast).", + "layers": [ + { + "id": "stage1", + "cwd": ".", + "parser": "python-fail", + "command": [ + "bash", + "tests/stage1_campaign_layer.sh" + ] + } + ], + "layers_comment": "One layer, and it BUILDS before it tests: the controls drive a compiled launcher, so a mutation to a .cs file would otherwise be invisible to them. The layer also sets OWEN_STAGE1_REQUIRE=1, so a control that cannot run is a failure rather than a silently shrinking denominator.", + "control": { + "id": "M00", + "description": "harness-honesty control: no mutation at all, which must report zero failing layers" + }, + "mutations": [ + { + "id": "M01", + "rule": "default-is-python", + "description": "the Stage-1 default engine is Rust — the cutover read as already decided, instead of Python remaining default until Gate G3", + "target": "frontend/roslyn/OwnSharp.Cli/EngineSelection.cs", + "pattern": "public const Engine Default = Engine\\.Python;", + "replacement": "public const Engine Default = Engine.Rust;", + "expected_catchers": [ + "stage1::default-stays-python" + ] + }, + { + "id": "M02", + "rule": "rust-run-needs-no-python", + "description": "Python is resolved for every engine — the old unconditional resolution kept 'just in case', which silently re-imposes a Python dependency on a Rust-only run", + "target": "frontend/roslyn/OwnSharp.Cli/EngineSelection.cs", + "pattern": "public static bool NeedsPython\\(Engine engine\\) => engine is Engine\\.Python or Engine\\.Compare;", + "replacement": "public static bool NeedsPython(Engine engine) => true;", + "expected_catchers": [ + "stage1::rust-actually-runs-rust" + ] + }, + { + "id": "M03", + "rule": "70-is-not-a-verdict", + "description": "70 is treated as a legal engine result — the shared internal-error code mistaken for part of the verdict contract because both engines document it", + "target": "frontend/roslyn/OwnSharp.Cli/EngineSelection.cs", + "pattern": "public static bool IsLegalEngineExit\\(int rc\\) => rc is 0 or 1 or 2;", + "replacement": "public static bool IsLegalEngineExit(int rc) => rc is 0 or 1 or 2 or 70;", + "expected_catchers": [ + "stage1::rc70-is-not-a-verdict" + ] + }, + { + "id": "M04", + "rule": "unexpected-rc-maps-to-5", + "description": "an unexpected Rust child status passes through as itself — 'propagate the child's exit code' read as faithfulness rather than as leaking a meaningless number to the caller", + "target": "frontend/roslyn/OwnSharp.Cli/CheckCommand.cs", + "pattern": "if \\(!EngineSelection\\.IsLegalEngineExit\\(rc\\)\\)", + "replacement": "if (rc == int.MinValue)", + "expected_catchers": [ + "stage1::unexpected-rc-maps-to-5", + "stage1::rc70-is-not-a-verdict", + "stage1::rust-failure-no-fallback" + ] + }, + { + "id": "M05", + "rule": "raw-rc-retained", + "description": "the raw child status is dropped from the report — the human-readable cause already names the number, so the typed carrier looks redundant", + "target": "frontend/roslyn/OwnSharp.Cli/CheckCommand.cs", + "pattern": "childExitCode: rc\\);", + "replacement": "childExitCode: null);", + "expected_catchers": [ + "stage1::raw-rc-retained" + ] + }, + { + "id": "M06", + "rule": "bad-locator-is-2-not-5", + "description": "an unusable OWEN_RUST_CORE is an internal error — a failure to start the engine read as Owen's own bug rather than the caller's configuration", + "target": "frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs", + "pattern": "public const int ExitCode = 2;", + "replacement": "public const int ExitCode = 5;", + "expected_catchers": [ + "stage1::bad-locator-is-2" + ] + }, + { + "id": "M07", + "rule": "bad-locator-is-2-not-3", + "description": "an unusable OWEN_RUST_CORE reuses exit 3 — 'no usable engine runtime' read as the same class as 'no usable Python', which it is not", + "target": "frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs", + "pattern": "public const int ExitCode = 2;", + "replacement": "public const int ExitCode = 3;", + "expected_catchers": [ + "stage1::bad-locator-is-2" + ] + }, + { + "id": "M08", + "rule": "no-fallback-in-the-shell", + "description": "the shell falls back to Python when the Rust core produces no verdict — 'be helpful, still give the user an answer', which is precisely the silent fallback every ruling forbids", + "target": "scripts/own-check.sh", + "pattern": " echo \\\"own-check: the Rust analysis core exited \\$rc, which is not a verdict \\(raw child status: \\$rc\\)\\. Owen did not fall back to Python\\.\\\" >&2\\n exit 5", + "replacement": " PYTHONPATH=\"$root\" python -m ownlang ownir \"$facts\" --format \"$format\" --severity \"$severity\"\n rc=$?", + "expected_catchers": [ + "stage1::rust-failure-no-fallback" + ] + }, + { + "id": "M09", + "rule": "shell-locator-is-2-not-3", + "description": "the shell reports an unusable OWEN_RUST_CORE as exit 3 — the Python-specific 'no usable runtime' code borrowed for the Rust candidate", + "target": "scripts/own-check.sh", + "pattern": " exit 2\\n fi\\nfi", + "replacement": " exit 3\n fi\nfi", + "expected_catchers": [ + "stage1::bad-locator-is-2" + ] + }, + { + "id": "M10", + "rule": "divergence-is-5", + "description": "a divergence exposes the reference's result — 'Python is still the reference, so trust it' read as a licence to answer while the two engines disagree", + "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", + "pattern": " return Fail\\(args, rust, py, rs,\\n \\$\\\"engine divergence", + "replacement": " await ReplayAsync(py).ConfigureAwait(false);\n return failOnFinding ? py.Rc : (py.Rc >= 2 ? py.Rc : 0);\n#pragma warning disable CS0162\n return Fail(args, rust, py, rs,\n $\"engine divergence", + "expected_catchers": [ + "stage1::divergence-is-5", + "stage1::compare-no-substitution" + ] + }, + { + "id": "M11", + "rule": "exec-failure-is-not-agreement", + "description": "the execution-failure check is skipped — comparing the results first and treating a crashed engine as just another difference, which loses the distinction D4.1 draws between (b) and (c)", + "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", + "pattern": " if \\(!pyLegal \\|\\| !rsLegal\\)", + "replacement": " if (false)", + "expected_catchers": [ + "stage1::exec-failure-is-5" + ] + }, + { + "id": "M12", + "rule": "zero-document-compare-fails", + "description": "a document with no analysable unit is compared anyway — 'both engines agreed' read as a result rather than as a zero denominator", + "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", + "pattern": " if \\(!HasAnalyzableUnit\\(captured, out var why\\)\\)", + "replacement": " if (false && !HasAnalyzableUnit(captured, out var why))", + "expected_catchers": [ + "stage1::compare-zero-document" + ] + }, + { + "id": "M13", + "rule": "candidate-identity-recorded", + "description": "the evidence records a placeholder candidate digest — the path already names the binary, so hashing it looks like belt-and-braces", + "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", + "pattern": "rust_core = new \\{ path = rust\\.Path, sha256 = rust\\.Sha256, bytes = rust\\.ByteLength \\},", + "replacement": "rust_core = new { path = rust.Path, sha256 = \"(not recorded)\", bytes = rust.ByteLength },", + "expected_catchers": [ + "stage1::candidate-identity" + ] + }, + { + "id": "M14", + "rule": "shell-compare-checks-stdout", + "description": "the shell compare stops comparing stdout — the exit code read as the whole of 'the public result', dropping the bytes the user actually sees", + "target": "scripts/own-check.sh", + "pattern": " cmp -s \"\\$cmp_dir/python\\.out\" \"\\$cmp_dir/rust\\.out\" \\|\\| diverged=\"\\$\\{diverged:\\+\\$diverged, \\}stdout\"", + "replacement": " true", + "expected_catchers": [ + "stage1::divergence-is-5" + ] + }, + { + "id": "M15", + "rule": "shell-divergence-is-5", + "description": "the shell reports a divergence as exit 1 — the 'something is wrong' tier reached for, when in public Owen 1 already means findings", + "target": "scripts/own-check.sh", + "pattern": " trap \\x27rm -f \"\\$facts\"\\x27 EXIT\\n exit 5", + "replacement": " trap 'rm -f \"$facts\"' EXIT\n exit 1", + "expected_catchers": [ + "stage1::divergence-is-5" + ] + }, + { + "id": "M16", + "rule": "shell-zero-document-fails", + "description": "the shell's zero-document guard is dropped — an empty document read as a legitimately clean agreement", + "target": "scripts/own-check.sh", + "pattern": " echo \"own-check: --engine compare: the captured OwnIR contains nothing to analyse — a compare over zero documents proves nothing and is a failure, not an agreement\\.\" >&2\\n exit 5", + "replacement": " :", + "expected_catchers": [ + "stage1::compare-zero-document" + ] + } + ] +} diff --git a/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs b/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs index 2b498972..e239c73c 100644 --- a/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs +++ b/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs @@ -36,6 +36,7 @@ public static async Task RunAsync(string[] args) { string format; string severity; + string engineName; bool failOnFinding; bool legacy; bool stats; @@ -44,7 +45,8 @@ public static async Task RunAsync(string[] args) List paths; try { - (format, severity, failOnFinding, legacy, stats, bodyThrowEdges, emitFacts, paths) = ParseArgs(args); + (format, severity, engineName, failOnFinding, legacy, stats, bodyThrowEdges, emitFacts, paths) = + ParseArgs(args); } catch (InvalidOperationException ex) { @@ -52,6 +54,15 @@ public static async Task RunAsync(string[] args) return 2; } + // #262 D1: the engine is selected HERE, from an explicit launcher + // selector, before anything engine-specific is resolved. An unknown + // engine is a usage error like any other. + if (!EngineSelection.TryParse(engineName, out var engine, out var engineError)) + { + Console.Error.WriteLine(engineError); + return 2; + } + if (!ValidFormats.Contains(format)) { Console.Error.WriteLine( @@ -78,16 +89,48 @@ public static async Task RunAsync(string[] args) return 4; } - // Resolve Python FIRST: no point extracting facts just to fail on stage 2. - ResolvedPython python; - try + // Engine-specific runtime resolution — AFTER the engine is chosen, and + // only for the engine actually chosen. Both resolutions happen before + // extraction for the same reason the Python one always did: no point + // extracting facts just to fail on stage 2. + // + // The asymmetry that used to live here is the whole finding this stage + // repairs: the launcher resolved Python and unpacked the vendored core + // unconditionally, because there was only ever one engine. A Rust-only + // run must do NEITHER — not as an optimisation, but because a Rust run + // that needs a working Python installation is not a Rust run, and it + // would quietly re-introduce the dependency the whole cutover exists to + // remove. + RustCore? rustCore = null; + if (EngineSelection.NeedsRust(engine)) { - python = PythonResolver.Resolve(); + try + { + rustCore = RustCoreLocator.Resolve(); + } + catch (RustCoreNotResolvedException ex) + { + // D3.1: a locator that cannot be used, detected before the + // selected Rust core has started, is a configuration error — + // exit 2. Never 3 (Python-specific), never 5 (our own bug), + // and never a fallback to Python. + Console.Error.WriteLine(ex.Message); + return RustCoreLocator.ExitCode; + } } - catch (PythonNotFoundException ex) + + ResolvedPython? python = null; + if (EngineSelection.NeedsPython(engine)) { - Console.Error.WriteLine(ex.Message); - return 3; + try + { + python = PythonResolver.Resolve(); + } + catch (PythonNotFoundException ex) + { + Console.Error.WriteLine(ex.Message); + return 3; + } } var factsPath = Path.GetTempFileName(); @@ -131,8 +174,32 @@ await RunExtractorAsync(paths, factsPath, legacy, stats, bodyThrowEdges) } } + // Stage 2 — the selected engine. Only the chosen branch touches + // its own runtime: `CoreVendor.EnsureUnpacked()` unpacks the + // vendored PYTHON core, so it lives inside the Python-needing + // branches and nowhere else. + if (engine == Engine.Rust) + { + var rustOutcome = await EngineRunner + .RunRustAsync(rustCore!, factsPath, format, severity, capture: false) + .ConfigureAwait(false); + return MapRustChildStatus(rustOutcome.Rc, args, failOnFinding); + } + var cacheRoot = CoreVendor.EnsureUnpacked(); - var rc = await RunCoreAsync(python, cacheRoot, factsPath, format, severity).ConfigureAwait(false); + + if (engine == Engine.Compare) + { + return await CompareMode + .RunAsync(python!, cacheRoot, rustCore!, factsPath, format, severity, + args, failOnFinding) + .ConfigureAwait(false); + } + + var pythonOutcome = await EngineRunner + .RunPythonAsync(python!, cacheRoot, factsPath, format, severity, capture: false) + .ConfigureAwait(false); + var rc = pythonOutcome.Rc; // The core self-reports internal errors as exit 70 (EX_SOFTWARE) // with one polite line (ownlang `run()`): surface them as OUR // internal error — pre-A1 a core crash exited 1 and, without @@ -159,11 +226,47 @@ await RunExtractorAsync(paths, factsPath, legacy, stats, bodyThrowEdges) } } - private static (string Format, string Severity, bool FailOnFinding, bool Legacy, bool Stats, - bool BodyThrowEdges, string? EmitFacts, List Paths) ParseArgs(string[] args) + /// + /// Map the Rust child's exit status onto Owen's public contract (#261's + /// launcher ruling, #262 D5). + /// + /// 0/1/2 are the engine's own verdict codes and pass through exactly + /// as the Python core's do. Everything else — the shared + /// internal-error code 70, a panic that escaped, a signal death, an + /// arbitrary 42 — is not a verdict, so it takes Owen's public + /// internal-error path (5) and the RAW child status is retained in the + /// diagnostic report's typed child_exit_code. An unexpected code + /// must never escape as itself: 42 read as an exit code is meaningless to + /// a caller, and 70 read as "findings" or "clean" is worse. + /// + /// No branch here runs Python. A Rust failure is a Rust failure. + /// + private static int MapRustChildStatus(int rc, string[] args, bool failOnFinding) + { + if (!EngineSelection.IsLegalEngineExit(rc)) + { + Console.Error.WriteLine( + $"owen: the Rust analysis core exited {rc}, which is not a verdict. " + + "Owen did not fall back to Python."); + return CrashReport.Child( + "analysis core (rust)", rc, args, capturedOutput: null, childExitCode: rc); + } + if (failOnFinding) + { + return rc; + } + return rc >= 2 ? rc : 0; + } + + private static (string Format, string Severity, string Engine, bool FailOnFinding, bool Legacy, + bool Stats, bool BodyThrowEdges, string? EmitFacts, List Paths) ParseArgs(string[] args) { var format = "human"; var severity = "error"; + // D1: Python is the Stage-1 default, and it is spelled here by asking + // EngineSelection rather than by writing "python" a second time — one + // place decides what the default engine is. + var engine = EngineSelection.ToName(EngineSelection.Default); var failOnFinding = false; var legacy = false; var stats = false; @@ -185,6 +288,7 @@ private static (string Format, string Severity, bool FailOnFinding, bool Legacy, case "--": onlyPaths = true; break; case "--format": format = RequireValue(args, ref i, "--format"); break; case "--severity": severity = RequireValue(args, ref i, "--severity"); break; + case "--engine": engine = RequireValue(args, ref i, "--engine"); break; case "--emit-facts": emitFacts = RequireValue(args, ref i, "--emit-facts"); break; case "--fail-on-finding": failOnFinding = true; break; case "--legacy": legacy = true; break; @@ -207,7 +311,7 @@ private static (string Format, string Severity, bool FailOnFinding, bool Legacy, } } - return (format, severity, failOnFinding, legacy, stats, bodyThrowEdges, emitFacts, paths); + return (format, severity, engine, failOnFinding, legacy, stats, bodyThrowEdges, emitFacts, paths); } private static string RequireValue(string[] args, ref int i, string flag) @@ -337,45 +441,4 @@ private static string ResolveDotnetMuxer() } return "dotnet"; } - - /// Stage 2: the one checker, run against the vendored core via the - /// resolved system Python. Findings print to the real stdout/stderr — this - /// is the surface the user actually asked for. - private static async Task RunCoreAsync( - ResolvedPython python, string cacheRoot, string factsPath, string format, string severity) - { - var psi = new ProcessStartInfo(python.FileName) - { - UseShellExecute = false, - WorkingDirectory = cacheRoot, - }; - foreach (var a in python.LeadingArgs) - { - psi.ArgumentList.Add(a); - } - psi.ArgumentList.Add("-m"); - psi.ArgumentList.Add("ownlang"); - psi.ArgumentList.Add("ownir"); - psi.ArgumentList.Add(factsPath); - psi.ArgumentList.Add("--format"); - psi.ArgumentList.Add(format); - psi.ArgumentList.Add("--severity"); - psi.ArgumentList.Add(severity); - // Belt-and-suspenders alongside WorkingDirectory: `-m` already adds the - // cwd to sys.path[0], but own-check.sh/.ps1 both set PYTHONPATH - // explicitly too, and matching that is cheap insurance. - psi.EnvironmentVariables["PYTHONPATH"] = cacheRoot; - // Debug passthrough (A1): the core's catch-all (`ownlang.run`) prints - // one polite line and exits 70; with OWNLANG_DEBUG=1 it re-raises the - // full traceback instead — that is what `owen check --debug` asks for. - if (CrashReport.Debug) - { - psi.EnvironmentVariables["OWNLANG_DEBUG"] = "1"; - } - - using var proc = Process.Start(psi) - ?? throw new InvalidOperationException("owen: failed to start the Python core process"); - await proc.WaitForExitAsync().ConfigureAwait(false); - return proc.ExitCode; - } } diff --git a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs new file mode 100644 index 00000000..1a2f5e32 --- /dev/null +++ b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs @@ -0,0 +1,339 @@ +using System.Security.Cryptography; +using System.Text.Json; + +namespace OwnSharp.Cli; + +/// +/// D4 / D4.1 — `--engine compare`: run the reference and the candidate over +/// ONE captured input and compare their public results. +/// +/// What it is. A launcher production-seam compare mode for +/// Stage-1/2 dogfood and CI. It is not a promised public feature yet, +/// and nothing about it is a fallback: neither engine's answer ever stands in +/// for the other's failure. +/// +/// Same input, proved. #260's load-bearing rule is that compare +/// evidence is never manufactured from two independent extractions. The +/// extractor runs once; its OwnIR bytes are read once into a single value; and +/// each engine's input file is materialised from that value and then +/// re-hashed and checked against it before any engine starts. "Both read the +/// same path" is an assumption; a recorded digest per engine, equal to the +/// capture's, is a measurement — and it is the measurement that makes "compare +/// fed the engines different bytes and still reported agreement" a control +/// that can actually go red. +/// +/// D4.1 — the result contract. Agreement exposes the +/// Python/reference result (Python is still default and reference in Stage 1). +/// Engine divergence and compare execution failure BOTH exit 5 with one +/// actionable stderr diagnostic plus reproduction evidence. Exit 5 is Owen's +/// internal-failure path: a reference-vs-candidate disagreement during an +/// explicit migration compare means Owen cannot honestly emit one answer, and +/// exit 1 is unavailable because in public Owen it already means findings. +/// +/// Known difference, not a defect. On native Windows the Python +/// reference encodes piped output as cp1252 with CRLF while the Rust candidate +/// emits canonical UTF-8 (#262's Windows A/B/C). Compare will therefore report +/// a real divergence there for non-ASCII output. That is the declared +/// behaviour change being visible, which is the point of measuring bytes. +/// +internal static class CompareMode +{ + /// D4.1: divergence and execution failure both take Owen's + /// public internal-error path. + public const int ExitCode = CrashReport.ExitCode; + + /// Run both engines over one capture and apply D4.1. + /// The public exit code: the reference's own on agreement, else 5. + public static async Task RunAsync( + ResolvedPython python, string cacheRoot, RustCore rust, string factsPath, + string format, string severity, string[] args, bool failOnFinding) + { + // --- capture ONCE ------------------------------------------------- + // One read of the extractor's output, one value, one identity. Every + // engine input below is derived from THIS array and nothing else. + // + // The digest is declared before the read so the failure paths below + // can record evidence even when there is no capture to hash; "" is + // "no capture was taken", which is exactly what that evidence says. + var capturedSha = ""; + byte[] captured; + try + { + captured = await File.ReadAllBytesAsync(factsPath).ConfigureAwait(false); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + return Fail(args, rust, null, null, + $"could not capture the extracted OwnIR: {ex.Message}", childExitCode: null); + } + + capturedSha = Sha256Hex(captured); + + // --- the zero-document guard -------------------------------------- + // A compare that judged nothing agrees about nothing. #260's rule that + // a run comparing zero documents FAILS applies here at the launcher's + // granularity: a document with no analyzable unit gives both engines + // nothing to disagree about, so a green result from it would be a + // zero denominator wearing a passing grade. + if (!HasAnalyzableUnit(captured, out var why)) + { + return Fail(args, rust, null, null, + $"the captured OwnIR contains nothing to analyse ({why}) — a compare over " + + "zero documents proves nothing and is a failure, not an agreement", + childExitCode: null); + } + + // --- materialise one input per engine, then PROVE they match ------- + var pythonInput = factsPath + ".compare-python.json"; + var rustInput = factsPath + ".compare-rust.json"; + try + { + await File.WriteAllBytesAsync(pythonInput, captured).ConfigureAwait(false); + await File.WriteAllBytesAsync(rustInput, captured).ConfigureAwait(false); + + var pythonSha = Sha256Hex(await File.ReadAllBytesAsync(pythonInput).ConfigureAwait(false)); + var rustSha = Sha256Hex(await File.ReadAllBytesAsync(rustInput).ConfigureAwait(false)); + if (pythonSha != capturedSha || rustSha != capturedSha) + { + return Fail(args, rust, null, null, + "the two engine inputs are not byte-identical to the single capture " + + $"(capture {capturedSha}, python {pythonSha}, rust {rustSha}) — the " + + "same-input invariant failed, so no comparison may be reported", + childExitCode: null); + } + + // --- run both ------------------------------------------------- + EngineOutcome py, rs; + try + { + py = await EngineRunner + .RunPythonAsync(python, cacheRoot, pythonInput, format, severity, capture: true) + .ConfigureAwait(false); + } + catch (Exception ex) when (ex is InvalidOperationException or IOException) + { + return Fail(args, rust, null, null, + $"the Python reference could not be run: {ex.Message}", childExitCode: null); + } + try + { + rs = await EngineRunner + .RunRustAsync(rust, rustInput, format, severity, capture: true) + .ConfigureAwait(false); + } + catch (Exception ex) when (ex is InvalidOperationException or IOException) + { + return Fail(args, rust, py, null, + $"the Rust candidate could not be run: {ex.Message}", childExitCode: null); + } + + // --- D4.1 (c): execution failure ------------------------------ + // Either engine failing to produce a verdict is an EXECUTION + // failure, never "the other engine's answer". Checked before + // divergence: two results are only comparable once both exist. + var pyLegal = EngineSelection.IsLegalEngineExit(py.Rc); + var rsLegal = EngineSelection.IsLegalEngineExit(rs.Rc); + if (!pyLegal || !rsLegal) + { + // The raw Rust child status is retained per D5 whenever the + // Rust side is the one that misbehaved. + int? childExit = rsLegal ? null : rs.Rc; + var offender = !pyLegal && !rsLegal + ? $"both engines failed (python exit {py.Rc}, rust exit {rs.Rc})" + : !pyLegal + ? $"the Python reference failed (exit {py.Rc})" + : $"the Rust candidate failed (exit {rs.Rc})"; + return Fail(args, rust, py, rs, + $"compare execution failure — {offender}. No engine's result was " + + "substituted for the other's failure.", + childExit); + } + + // --- D4.1 (a)/(b): agreement or divergence -------------------- + var sameOut = py.Stdout.AsSpan().SequenceEqual(rs.Stdout); + var sameErr = py.Stderr.AsSpan().SequenceEqual(rs.Stderr); + var sameRc = py.Rc == rs.Rc; + if (!sameOut || !sameErr || !sameRc) + { + var what = string.Join(", ", new[] + { + sameRc ? null : $"exit ({py.Rc} vs {rs.Rc})", + sameOut ? null : $"stdout ({py.Stdout.Length} vs {rs.Stdout.Length} bytes)", + sameErr ? null : $"stderr ({py.Stderr.Length} vs {rs.Stderr.Length} bytes)", + }.Where(x => x is not null)); + return Fail(args, rust, py, rs, + $"engine divergence — the reference and the candidate disagree on {what}. " + + "Neither verdict is exposed as authoritative: Owen cannot honestly emit " + + "one answer when its reference and its candidate disagree.", + childExitCode: null); + } + + // --- agreement: the externally observed result is the reference's + // Stage 1 keeps Python as default AND reference, so on agreement + // the user sees exactly what a `--engine python` run would have + // produced — byte for byte, replayed undecoded. + await ReplayAsync(py).ConfigureAwait(false); + WriteEvidence(args, rust, capturedSha, py, rs, verdict: "agreement", + diagnostic: null, childExitCode: null); + return failOnFinding ? py.Rc : (py.Rc >= 2 ? py.Rc : 0); + } + finally + { + TryDelete(pythonInput); + TryDelete(rustInput); + } + + int Fail(string[] a, RustCore core, EngineOutcome? py, EngineOutcome? rs, + string diagnostic, int? childExitCode) + { + Console.Error.WriteLine($"owen: --engine compare: {diagnostic}"); + var path = WriteEvidence(a, core, capturedSha, py, rs, + verdict: childExitCode is null && py is not null && rs is not null + ? "divergence" + : "execution-failure", + diagnostic: diagnostic, childExitCode: childExitCode); + if (path is not null) + { + Console.Error.WriteLine($" Reproduction evidence: {path}"); + } + return ExitCode; + } + } + + /// Replay the reference's captured streams to the real ones, + /// undecoded — what the user would have seen without the compare. + private static async Task ReplayAsync(EngineOutcome reference) + { + if (reference.Stdout.Length > 0) + { + await using var stdout = Console.OpenStandardOutput(); + await stdout.WriteAsync(reference.Stdout).ConfigureAwait(false); + await stdout.FlushAsync().ConfigureAwait(false); + } + if (reference.Stderr.Length > 0) + { + await using var stderr = Console.OpenStandardError(); + await stderr.WriteAsync(reference.Stderr).ConfigureAwait(false); + await stderr.FlushAsync().ConfigureAwait(false); + } + } + + /// + /// Does this OwnIR document carry anything an engine could analyse? + /// + /// The schema requires only ownir_version and module; + /// everything analysable lives in the optional collections. All of them + /// empty or absent means there is no case to compare. + /// + private static bool HasAnalyzableUnit(byte[] ownir, out string why) + { + if (ownir.Length == 0) + { + why = "the capture is empty"; + return false; + } + string[] collections = + ["components", "functions", "services", "effects", "protocols", "protocol_functions"]; + try + { + using var doc = JsonDocument.Parse(ownir); + if (doc.RootElement.ValueKind != JsonValueKind.Object) + { + why = "the capture is not an OwnIR object"; + return false; + } + foreach (var name in collections) + { + if (doc.RootElement.TryGetProperty(name, out var value) + && value.ValueKind == JsonValueKind.Array + && value.GetArrayLength() > 0) + { + why = ""; + return true; + } + } + why = "every OwnIR collection is empty or absent"; + return false; + } + catch (JsonException ex) + { + // Not our call to make: a document the strict door should refuse + // is a legitimate compare case (both engines must refuse it the + // same way), so an unparseable capture is NOT treated as + // zero-document here. Let both engines answer it. + why = ""; + _ = ex; + return true; + } + } + + /// One reproducible compare artifact per run, overwritten in + /// place. It records what ran, over which bytes, with which candidate + /// identity, and what each engine answered — enough to re-run the exact + /// comparison by hand. No source contents, like the crash report. + private static string? WriteEvidence( + string[] args, RustCore rust, string capturedSha, + EngineOutcome? py, EngineOutcome? rs, string verdict, + string? diagnostic, int? childExitCode) + { + try + { + var dir = Path.Combine( + Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), + ".owen", "compare"); + Directory.CreateDirectory(dir); + var path = Path.Combine(dir, "last-compare.json"); + var report = new + { + schema = 1, + tool = "owen", + version = ToolVersion.Current, + timestamp_utc = DateTime.UtcNow.ToString("o"), + mode = "compare", + verdict, + diagnostic, + args, + // The same-input attestation: one capture, and the digest both + // engine inputs were verified against before either started. + input = new { sha256 = capturedSha }, + // D3: the candidate's recorded identity, so a stale binary + // cannot stand in without the evidence changing. + rust_core = new { path = rust.Path, sha256 = rust.Sha256, bytes = rust.ByteLength }, + python = py is null ? null : new + { + exit = py.Rc, + stdout_sha256 = Sha256Hex(py.Stdout), + stderr_sha256 = Sha256Hex(py.Stderr), + stdout_bytes = py.Stdout.Length, + stderr_bytes = py.Stderr.Length, + }, + rust = rs is null ? null : new + { + exit = rs.Rc, + stdout_sha256 = Sha256Hex(rs.Stdout), + stderr_sha256 = Sha256Hex(rs.Stderr), + stdout_bytes = rs.Stdout.Length, + stderr_bytes = rs.Stderr.Length, + }, + // D5: the raw Rust child status, retained whenever the Rust + // child produced one outside the legal set. + child_exit_code = childExitCode, + }; + File.WriteAllText(path, JsonSerializer.Serialize( + report, new JsonSerializerOptions { WriteIndented = true })); + return path; + } + catch (Exception) + { + return null; + } + } + + private static string Sha256Hex(byte[] data) => + Convert.ToHexString(SHA256.HashData(data)).ToLowerInvariant(); + + private static void TryDelete(string path) + { + try { File.Delete(path); } catch (IOException) { /* best-effort cleanup */ } + } +} diff --git a/frontend/roslyn/OwnSharp.Cli/CrashReport.cs b/frontend/roslyn/OwnSharp.Cli/CrashReport.cs index 7d513ef1..a51855e9 100644 --- a/frontend/roslyn/OwnSharp.Cli/CrashReport.cs +++ b/frontend/roslyn/OwnSharp.Cli/CrashReport.cs @@ -45,7 +45,7 @@ public static int Handle(Exception ex, string[] args) } var report = TryWrite(args, stage: "owen", cause: $"{ex.GetType().FullName}: {ex.Message}", - detail: ex.ToString(), childOutput: null); + detail: ex.ToString(), childOutput: null, childExitCode: null); Console.Error.WriteLine($"owen: internal error ({ex.GetType().Name}: {ex.Message})"); Emit(report); return ExitCode; @@ -53,12 +53,20 @@ public static int Handle(Exception ex, string[] args) /// Frame a child stage's crash (unexpected exit code) without /// dumping its raw output on the user; the full capture goes into the - /// report instead. In debug mode the caller prints the raw output. - public static int Child(string stage, int rc, string[] args, string? capturedOutput) + /// report instead. In debug mode the caller prints the raw output. + /// + /// is #262's D5 carrier: the RAW + /// child status, retained as a typed integer in the report. The + /// human-readable cause below also mentions the number, but prose + /// is not evidence — a machine that must answer "what exactly did the + /// child exit with?" reads the typed field, and the forced-unexpected-rc + /// control asserts on that field rather than on a sentence. + public static int Child( + string stage, int rc, string[] args, string? capturedOutput, int? childExitCode = null) { var report = TryWrite(args, stage, cause: $"{stage} exited with unexpected code {rc}", - detail: null, childOutput: capturedOutput); + detail: null, childOutput: capturedOutput, childExitCode: childExitCode); Console.Error.WriteLine( $"owen: the {stage} stage failed internally (exit {rc})."); Emit(report); @@ -82,7 +90,8 @@ private static void Emit(string? reportPath) /// well-known path beats an ever-growing directory). Best-effort: a /// failure to write the report must never mask the original failure. private static string? TryWrite( - string[] args, string stage, string cause, string? detail, string? childOutput) + string[] args, string stage, string cause, string? detail, string? childOutput, + int? childExitCode) { try { @@ -93,7 +102,10 @@ private static void Emit(string? reportPath) var path = Path.Combine(dir, "last-failure.json"); var report = new { - schema = 1, + // Bumped to 2 by #262 D5: the report gained the typed + // `child_exit_code` field below. `stage` keeps identifying + // WHICH child; no second taxonomy was introduced for it. + schema = 2, tool = "owen", version = ToolVersion.Current, timestamp_utc = DateTime.UtcNow.ToString("o"), @@ -105,6 +117,11 @@ private static void Emit(string? reportPath) cause, detail, child_output = childOutput, + // D5: a typed nullable integer, null for every failure that is + // not a child's unexpected status. Never a string, never + // absent — a consumer must be able to distinguish "no child + // status" from "the child exited 0". + child_exit_code = childExitCode, }; File.WriteAllText(path, JsonSerializer.Serialize( report, new JsonSerializerOptions { WriteIndented = true })); diff --git a/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs b/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs new file mode 100644 index 00000000..c5ba4c06 --- /dev/null +++ b/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs @@ -0,0 +1,139 @@ +using System.Diagnostics; + +namespace OwnSharp.Cli; + +/// One engine's observable result: the public streams and the exit +/// code, exactly as the child produced them. +/// +/// The streams are RAW BYTES, not decoded text. Compare mode claims the +/// engines' public results are byte-identical, and a claim about bytes cannot +/// be measured on strings: decoding both children through one .NET encoding +/// would silently normalise away a real difference (the Windows A/B/C +/// cp1252-vs-UTF-8 behaviour change is exactly such a case) or invent one from +/// a replacement character. Bytes in, bytes compared, bytes replayed. +/// +/// Streams are captured only when the caller asked for capture (compare +/// mode); otherwise they went straight to the real stdout/stderr and the +/// arrays are empty. +/// The child's raw exit code, unmapped. +/// Captured stdout bytes, or empty when not capturing. +/// Captured stderr bytes, or empty when not capturing. +internal sealed record EngineOutcome(int Rc, byte[] Stdout, byte[] Stderr); + +/// +/// Runs one analysis engine over one OwnIR facts file. +/// +/// Both engines are invoked at the same seam with the same arguments, +/// which is the point: the compare mode below can only be honest if the two +/// runs differ in the engine and in nothing else. The Rust side invokes the +/// PRODUCTION binary `own-cli ownir` — never `own-shadow-engine`, which is +/// #260's dev oracle and is not wired into production by this stage or any +/// other. +/// +internal static class EngineRunner +{ + /// Stage 2, the Python reference: the vendored core run by the + /// resolved system Python. Unchanged from the pre-Stage-1 launcher except + /// that it can now be asked to capture its streams for compare mode. + public static async Task RunPythonAsync( + ResolvedPython python, string cacheRoot, string factsPath, + string format, string severity, bool capture) + { + var psi = new ProcessStartInfo(python.FileName) + { + UseShellExecute = false, + WorkingDirectory = cacheRoot, + }; + foreach (var a in python.LeadingArgs) + { + psi.ArgumentList.Add(a); + } + psi.ArgumentList.Add("-m"); + psi.ArgumentList.Add("ownlang"); + psi.ArgumentList.Add("ownir"); + psi.ArgumentList.Add(factsPath); + psi.ArgumentList.Add("--format"); + psi.ArgumentList.Add(format); + psi.ArgumentList.Add("--severity"); + psi.ArgumentList.Add(severity); + // Belt-and-suspenders alongside WorkingDirectory: `-m` already adds the + // cwd to sys.path[0], but own-check.sh/.ps1 both set PYTHONPATH + // explicitly too, and matching that is cheap insurance. + psi.EnvironmentVariables["PYTHONPATH"] = cacheRoot; + // Debug passthrough (A1): the core's catch-all (`ownlang.run`) prints + // one polite line and exits 70; with OWNLANG_DEBUG=1 it re-raises the + // full traceback instead — that is what `owen check --debug` asks for. + if (CrashReport.Debug) + { + psi.EnvironmentVariables["OWNLANG_DEBUG"] = "1"; + } + + return await RunAsync(psi, capture, "the Python core").ConfigureAwait(false); + } + + /// + /// Stage 2, the Rust candidate: the production executable `own-cli ownir`. + /// + /// The argument vector mirrors the Python invocation exactly — same + /// facts path, same --format, same --severity — because + /// #261 built this binary to reproduce that contract. No engine flag is + /// passed: `own-cli` presents one engine and knows nothing of Python + /// (C-4), and Stage 1 does not change that by handing it a selector. + /// + public static async Task RunRustAsync( + RustCore core, string factsPath, string format, string severity, bool capture) + { + var psi = new ProcessStartInfo(core.Path) + { + UseShellExecute = false, + }; + psi.ArgumentList.Add("ownir"); + psi.ArgumentList.Add(factsPath); + psi.ArgumentList.Add("--format"); + psi.ArgumentList.Add(format); + psi.ArgumentList.Add("--severity"); + psi.ArgumentList.Add(severity); + if (CrashReport.Debug) + { + psi.EnvironmentVariables["OWNLANG_DEBUG"] = "1"; + } + + return await RunAsync(psi, capture, "the Rust core").ConfigureAwait(false); + } + + /// Start the child and collect its result. Capturing drains both + /// pipes concurrently AS BYTES (a serial read deadlocks once either pipe + /// fills, and a decoded read would not answer a byte question); not + /// capturing leaves the child attached to the real streams so the user + /// sees the engine's own output live, exactly as before Stage 1. + private static async Task RunAsync( + ProcessStartInfo psi, bool capture, string what) + { + psi.RedirectStandardOutput = capture; + psi.RedirectStandardError = capture; + + using var proc = Process.Start(psi) + ?? throw new InvalidOperationException($"owen: failed to start {what} process"); + + if (!capture) + { + await proc.WaitForExitAsync().ConfigureAwait(false); + return new EngineOutcome(proc.ExitCode, [], []); + } + + var stdoutTask = DrainAsync(proc.StandardOutput.BaseStream); + var stderrTask = DrainAsync(proc.StandardError.BaseStream); + await proc.WaitForExitAsync().ConfigureAwait(false); + var stdout = await stdoutTask.ConfigureAwait(false); + var stderr = await stderrTask.ConfigureAwait(false); + return new EngineOutcome(proc.ExitCode, stdout, stderr); + } + + /// Read one redirected pipe to end, undecoded. + private static async Task DrainAsync(Stream stream) + { + using var buffer = new MemoryStream(); + await stream.CopyToAsync(buffer).ConfigureAwait(false); + return buffer.ToArray(); + } +} diff --git a/frontend/roslyn/OwnSharp.Cli/EngineSelection.cs b/frontend/roslyn/OwnSharp.Cli/EngineSelection.cs new file mode 100644 index 00000000..5ae8f130 --- /dev/null +++ b/frontend/roslyn/OwnSharp.Cli/EngineSelection.cs @@ -0,0 +1,106 @@ +namespace OwnSharp.Cli; + +/// +/// Which analysis engine `owen check` runs (#262 D1). The selector belongs to +/// the LAUNCHER and only to the launcher: `own-cli` presents one engine and +/// knows nothing of Python (#261 C-4), so nothing here is ever forwarded to it +/// as an engine flag. +/// +internal enum Engine +{ + /// The vendored Python core. Stage-1 default and the reference. + Python, + + /// The Rust core (`own-cli ownir`), opt-in at Stage 1. + Rust, + + /// Both engines over one captured input, compared (D4/D4.1). + Compare, +} + +/// +/// Parsing and the shared exit-code contract for the selected engine. +/// +/// Python is the Stage-1 default (D1) and this type is where that +/// is written down once: is the single place a reader — +/// or a mutation — can move it, which is what makes the "the default silently +/// became Rust" control load-bearing rather than a matter of reading four +/// launchers and hoping. +/// +internal static class EngineSelection +{ + /// D1: Python remains the default for the whole of Stage 1. The + /// public default does not move before Gate G3 (#262 Stage 3). + public const Engine Default = Engine.Python; + + /// The spellings accepted by every launcher surface. One contract + /// across `owen`, own-check.sh, own-check.ps1 and the Action (D2). + public static readonly string[] Names = ["python", "rust", "compare"]; + + /// + /// The exit codes an engine may legitimately produce for a document it + /// actually judged: 0 clean, 1 findings, 2 usage error or a refusal by the + /// strict door. Anything else is NOT a verdict. + /// + /// 70 is deliberately NOT in this set. It is the engines' shared + /// internal-error code (`ownlang.run()` and `own-cli` both use it), so it + /// is a KNOWN failure rather than an unexpected status — but it is still a + /// failure, and reading it as a finding or as clean is exactly the bug + /// control (6) exists to catch. It maps to Owen's public internal-error + /// path (5) like any other non-verdict, and its raw value is retained in + /// the report the same way. + /// + public static bool IsLegalEngineExit(int rc) => rc is 0 or 1 or 2; + + /// The engines' shared internal-error code. Known, still a + /// failure — never a verdict. + public const int EngineInternalError = 70; + + public static string ToName(Engine engine) => engine switch + { + Engine.Python => "python", + Engine.Rust => "rust", + Engine.Compare => "compare", + _ => "python", + }; + + /// Parse a --engine value. Returns false (with an + /// actionable message) rather than throwing, so every surface answers an + /// unknown engine as a usage error (exit 2) in one voice. + public static bool TryParse(string value, out Engine engine, out string error) + { + switch (value) + { + case "python": + engine = Engine.Python; + error = ""; + return true; + case "rust": + engine = Engine.Rust; + error = ""; + return true; + case "compare": + engine = Engine.Compare; + error = ""; + return true; + default: + engine = Default; + error = + $"owen check: unknown --engine '{value}' (choose: {string.Join(", ", Names)})"; + return false; + } + } + + /// True when the selection needs the Rust candidate binary — and + /// therefore needs OWEN_RUST_CORE to resolve (D3/D3.1). + public static bool NeedsRust(Engine engine) => engine is Engine.Rust or Engine.Compare; + + /// True when the selection needs the Python reference — and + /// therefore may resolve an interpreter and unpack the vendored core. + /// + /// This is the predicate the "a Rust-only run must not resolve or + /// unpack Python" finding turns on. It is written as its own function, not + /// inlined at the call site, precisely so a mutation that widens it (say, + /// back to "always") has one obvious place to be caught. + public static bool NeedsPython(Engine engine) => engine is Engine.Python or Engine.Compare; +} diff --git a/frontend/roslyn/OwnSharp.Cli/Program.cs b/frontend/roslyn/OwnSharp.Cli/Program.cs index ce1cef0a..8513ee32 100644 --- a/frontend/roslyn/OwnSharp.Cli/Program.cs +++ b/frontend/roslyn/OwnSharp.Cli/Program.cs @@ -61,6 +61,7 @@ owen check [more paths...] [options] Options (mirrors scripts/own-check.sh): --format {human|github|msbuild|sarif} finding surface (default: human) --severity {error|warning} how findings are shown (default: error) + --engine {python|rust|compare} analysis engine (default: python) --fail-on-finding exit with the core's code (1 = findings) instead of always 0 --emit-facts also write the intermediate OwnIR facts.json here --legacy use the flat name-based local-IDisposable detector @@ -81,6 +82,16 @@ 4 no supported input found (never a silent clean scan) fallback), else `py -3` (Windows) / `python3` (elsewhere); must be >=3.11. No auto-install — see the error message if none is found. + Engine (#262 Stage 1): Python is the default and the reference. `rust` + runs the Rust core instead; `compare` runs both over one captured input + and reports the reference's result only when they agree byte for byte. + Rust and compare need the candidate binary's absolute path in + OWEN_RUST_CORE — there is NO discovery (no PATH lookup, no target/ + probing), so a missing or unusable OWEN_RUST_CORE is a configuration + error (exit 2), never a silent fall back to Python. A Rust failure is + never turned into a Python success in any mode. `compare` is a + development/CI seam for the migration, not yet a promised feature. + Input that doesn't match the included frontend (e.g. no .cs/.csproj/.sln found anywhere given) fails explicitly (exit 4) rather than reporting a clean scan. diff --git a/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs new file mode 100644 index 00000000..f37373f0 --- /dev/null +++ b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs @@ -0,0 +1,146 @@ +using System.Security.Cryptography; + +namespace OwnSharp.Cli; + +/// Thrown when the Stage-1 Rust candidate cannot be resolved. The +/// message is already the full one-line, actionable text; the caller turns it +/// into the ratified exit code and prints nothing of its own. +internal sealed class RustCoreNotResolvedException(string message) : Exception(message); + +/// The resolved candidate binary and its recorded identity. +/// Absolute path to the `own-cli` binary that will run. +/// Lowercase hex SHA-256 of the file that was resolved. +/// Length in bytes of that same file. +internal sealed record RustCore(string Path, string Sha256, long ByteLength); + +/// +/// D3 — the Stage-1 Rust candidate locator. +/// +/// The selector and the binary's location are different concepts. +/// --engine says WHICH engine; OWEN_RUST_CORE says WHERE the +/// candidate `own-cli` binary is. The ratified development locator is exactly +/// OWEN_RUST_CORE — an absolute path to the binary — and no alternate +/// spelling remains open. +/// +/// No discovery of any kind. No PATH lookup, no +/// rust/target/{debug,release} probing, no "first binary found". Those +/// are how a stale binary silently stands in for the one under test, and D3 +/// closes that door: the caller says exactly which file to run, or the run +/// fails visibly. Stage 3's packaged resolution is D6's problem, not this +/// one's — do not solve packaging here. +/// +/// D3.1 — the exit code. For --engine rust|compare, a +/// missing, empty, nonexistent, non-file or non-executable +/// OWEN_RUST_CORE, detected BEFORE the selected Rust core has +/// successfully started, is a launcher configuration/contract error: public +/// exit 2, with one actionable diagnostic. It is not 3 (that is +/// Python-specific), not 5 (that is Owen's own internal failure), and it never +/// triggers a Python fallback. Once the candidate has actually spawned, its +/// status is governed by the Rust-child / D4.1 / D5 rules instead — that seam +/// is exactly what the Stage-1 controls mutate. +/// +internal static class RustCoreLocator +{ + /// The one ratified Stage-1 development locator (D3). Named once + /// so no second spelling can quietly appear beside it. + public const string EnvVar = "OWEN_RUST_CORE"; + + /// Public exit code for an unusable locator (D3.1). A launcher + /// configuration error, in the same tier as any other usage mistake. + public const int ExitCode = 2; + + /// Resolve the candidate, or throw with the actionable reason. + /// Every rejection is a configuration/usage failure, never a fallback. + public static RustCore Resolve() + { + var raw = Environment.GetEnvironmentVariable(EnvVar); + + if (raw is null) + { + throw new RustCoreNotResolvedException(Problem("is not set")); + } + if (string.IsNullOrWhiteSpace(raw)) + { + throw new RustCoreNotResolvedException(Problem("is set but empty")); + } + + // A directory that exists still fails the File.Exists test below, but + // saying "is a directory" beats saying "does not exist" about a path + // the user can see with their own eyes. + if (Directory.Exists(raw)) + { + throw new RustCoreNotResolvedException( + Problem($"points at a directory, not a file: '{raw}'")); + } + if (!File.Exists(raw)) + { + throw new RustCoreNotResolvedException( + Problem($"points at a path that does not exist: '{raw}'")); + } + if (!IsExecutable(raw)) + { + throw new RustCoreNotResolvedException( + Problem($"points at a file that is not executable: '{raw}'")); + } + + string sha; + long length; + try + { + using var stream = File.OpenRead(raw); + length = stream.Length; + sha = Convert.ToHexString(SHA256.HashData(stream)).ToLowerInvariant(); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + throw new RustCoreNotResolvedException( + Problem($"points at a file that cannot be read: '{raw}' ({ex.Message})")); + } + + return new RustCore(Path.GetFullPath(raw), sha, length); + } + + /// One diagnostic shape for every rejection, so the reason varies + /// and the contract does not. + private static string Problem(string what) => + $"owen check: --engine rust/compare needs the candidate `own-cli` binary, but " + + $"{EnvVar} {what}. Set {EnvVar} to the absolute path of the `own-cli` " + + $"executable to run (Stage 1 does no discovery: no PATH lookup, no " + + $"rust/target probing). Owen did not fall back to Python — an explicitly " + + $"selected engine that cannot be started is a configuration error, not a " + + $"reason to run something else."; + + /// + /// Is this file runnable as a program? + /// + /// On Unix that is a real permission question, so it is asked of the + /// file mode: any of the three execute bits. On Windows there is no + /// execute bit — runnability is decided by the loader — so an existing + /// regular file is accepted and a genuinely broken image fails later, at + /// spawn, which is the D3.1 seam's other side and already maps to the + /// internal-error path. + /// + private static bool IsExecutable(string path) + { + if (OperatingSystem.IsWindows()) + { + return true; + } + try + { + var mode = File.GetUnixFileMode(path); + return (mode & (UnixFileMode.UserExecute + | UnixFileMode.GroupExecute + | UnixFileMode.OtherExecute)) != 0; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException + or PlatformNotSupportedException) + { + // Cannot read the mode: refuse rather than assume runnable. A + // false "yes" here would turn a configuration error into a spawn + // failure reported as an internal error, which loses D3.1's + // distinction between "could not select" and "ran and misbehaved". + return false; + } + } +} diff --git a/scripts/own-check.ps1 b/scripts/own-check.ps1 index 9140a611..c77b79be 100644 --- a/scripts/own-check.ps1 +++ b/scripts/own-check.ps1 @@ -21,6 +21,15 @@ .PARAMETER Severity How a host shows findings: error (default) or warning (advisory). +.PARAMETER Engine + Which analysis engine runs (#262 Stage 1): python (DEFAULT and reference), + rust (the Rust core `own-cli ownir`), or compare (both over one captured + input, exposing the reference's result only when they agree byte for byte). + rust and compare require the candidate binary's absolute path in + OWEN_RUST_CORE — there is no discovery of any kind, so an unset or unusable + OWEN_RUST_CORE is a configuration error (exit 2), never a silent fall back to + Python. A Rust failure is never turned into a Python success in any mode. + .PARAMETER Verbosity How much to print: quiet (errors only — hide the advisory OWN050 "leakage analysis skipped" notes, P-014 Tier A), normal (default), or verbose (also a @@ -49,6 +58,11 @@ param( [string]$Root, [string]$Format = "human", [string]$Severity = "error", + # D1: Python is the Stage-1 default on every launcher surface. ValidateSet + # makes an unknown engine a parameter-binding failure rather than a value + # that reaches the dispatch below. + [ValidateSet("python", "rust", "compare")] + [string]$Engine = "python", [ValidateSet("quiet", "normal", "verbose")] [string]$Verbosity = "normal", [switch]$Legacy, @@ -77,6 +91,36 @@ if ([string]::IsNullOrEmpty($Root)) { if ($Paths) { $Paths = @($Paths | Where-Object { $_ -ne "--" }) } if (-not $Paths -or $Paths.Count -eq 0) { $Paths = @(".") } +# D3/D3.1 — the Stage-1 Rust candidate locator, resolved BEFORE anything is +# extracted. OWEN_RUST_CORE is the one ratified spelling and there is NO +# discovery (no PATH lookup, no rust\target probing), because discovery is how +# a stale binary silently stands in for the one under test. Every rejection is +# a configuration error (exit 2) — not 3 (Python-specific), not 5 (an internal +# failure) — and none of them falls back to Python. +$rustCore = "" +if ($Engine -eq "rust" -or $Engine -eq "compare") { + $rustCore = $env:OWEN_RUST_CORE + $problem = "" + if ([string]::IsNullOrWhiteSpace($rustCore)) { + $problem = "is not set (or is empty)" + } + elseif (Test-Path -LiteralPath $rustCore -PathType Container) { + $problem = "points at a directory, not a file: '$rustCore'" + } + elseif (-not (Test-Path -LiteralPath $rustCore -PathType Leaf)) { + $problem = "points at a path that does not exist: '$rustCore'" + } + if ($problem -ne "") { + # Windows has no execute bit: an existing regular file is accepted here + # and a genuinely broken image fails at spawn, which is the other side + # of the D3.1 seam and already maps to the internal-error path. + Write-Error -Message ("own-check: --engine $Engine needs the candidate ``own-cli`` binary, but " + + "OWEN_RUST_CORE $problem. Set OWEN_RUST_CORE to the absolute path of the ``own-cli`` " + + "executable to run. Owen did not fall back to Python.") -ErrorAction Continue + exit 2 + } +} + $extractor = Join-Path $Root "frontend\roslyn\OwnSharp.Extractor" $facts = New-TemporaryFile try { @@ -97,12 +141,130 @@ try { exit $stage1 } - # Stage 2: the one checker produces the verdict at the C# location. + # Stage 2: the SELECTED engine produces the verdict at the C# location. $env:PYTHONPATH = $Root $ownirArgs = @($facts.FullName, "--format", $Format, "--severity", $Severity, "--verbosity", $Verbosity) - & python -m ownlang ownir @ownirArgs - $rc = $LASTEXITCODE + + if ($Engine -eq "python") { + & python -m ownlang ownir @ownirArgs + $rc = $LASTEXITCODE + } + elseif ($Engine -eq "rust") { + # The PRODUCTION Rust executable, never own-shadow-engine. + $rustArgs = @("ownir") + $ownirArgs + & $rustCore @rustArgs + $rc = $LASTEXITCODE + # 0/1/2 are verdicts and pass through; anything else is not a verdict + # and takes the public internal-error path (5) with the raw child + # status named. It never runs Python instead. + if ($rc -ne 0 -and $rc -ne 1 -and $rc -ne 2) { + Write-Error -Message ("own-check: the Rust analysis core exited $rc, which is not a " + + "verdict (raw child status: $rc). Owen did not fall back to Python.") -ErrorAction Continue + exit 5 + } + } + else { + # D4/D4.1 — both engines over ONE capture, proved byte-identical. + $cmpDir = Join-Path ([System.IO.Path]::GetTempPath()) ("owen-compare-" + [guid]::NewGuid().ToString("N")) + New-Item -ItemType Directory -Path $cmpDir | Out-Null + # Declared before the try so the finally can read it under + # Set-StrictMode -Version Latest, where touching an undefined variable + # is an error rather than $null. + $keep = $false + try { + $capture = Join-Path $cmpDir "capture.json" + Copy-Item -LiteralPath $facts.FullName -Destination $capture + $captureSha = (Get-FileHash -LiteralPath $capture -Algorithm SHA256).Hash.ToLowerInvariant() + + # A compare that judged nothing agrees about nothing. + $doc = $null + try { $doc = Get-Content -LiteralPath $capture -Raw -Encoding utf8 | ConvertFrom-Json } catch { $doc = $null } + if ($null -ne $doc) { + $hasUnit = $false + foreach ($k in @("components", "functions", "services", "effects", "protocols", "protocol_functions")) { + $v = $doc.PSObject.Properties[$k] + if ($null -ne $v -and $null -ne $v.Value -and @($v.Value).Count -gt 0) { $hasUnit = $true; break } + } + if (-not $hasUnit) { + Write-Error -Message ("own-check: --engine compare: the captured OwnIR contains nothing to " + + "analyse — a compare over zero documents proves nothing and is a failure, not an " + + "agreement.") -ErrorAction Continue + exit 5 + } + } + + $pyIn = Join-Path $cmpDir "python-input.json" + $rsIn = Join-Path $cmpDir "rust-input.json" + Copy-Item -LiteralPath $capture -Destination $pyIn + Copy-Item -LiteralPath $capture -Destination $rsIn + $pyInSha = (Get-FileHash -LiteralPath $pyIn -Algorithm SHA256).Hash.ToLowerInvariant() + $rsInSha = (Get-FileHash -LiteralPath $rsIn -Algorithm SHA256).Hash.ToLowerInvariant() + if ($pyInSha -ne $captureSha -or $rsInSha -ne $captureSha) { + Write-Error -Message ("own-check: --engine compare: the two engine inputs are not byte-identical " + + "to the single capture (capture $captureSha, python $pyInSha, rust $rsInSha) — the " + + "same-input invariant failed, so no comparison may be reported.") -ErrorAction Continue + exit 5 + } + + # Start-Process redirects the children's RAW bytes to files: a + # claim about byte-identical output cannot be measured through + # PowerShell's own string pipeline. + $pyArgs = @("-m", "ownlang", "ownir", $pyIn, "--format", $Format, "--severity", $Severity, + "--verbosity", $Verbosity) + $p1 = Start-Process -FilePath "python" -ArgumentList $pyArgs -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput (Join-Path $cmpDir "python.out") ` + -RedirectStandardError (Join-Path $cmpDir "python.err") + $rsArgs = @("ownir", $rsIn, "--format", $Format, "--severity", $Severity, + "--verbosity", $Verbosity) + $p2 = Start-Process -FilePath $rustCore -ArgumentList $rsArgs -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput (Join-Path $cmpDir "rust.out") ` + -RedirectStandardError (Join-Path $cmpDir "rust.err") + $pyRc = $p1.ExitCode + $rsRc = $p2.ExitCode + + # D4.1 (c): execution failure first — two results are comparable + # only once both exist. + $pyLegal = ($pyRc -eq 0 -or $pyRc -eq 1 -or $pyRc -eq 2) + $rsLegal = ($rsRc -eq 0 -or $rsRc -eq 1 -or $rsRc -eq 2) + if (-not $pyLegal -or -not $rsLegal) { + Write-Error -Message ("own-check: --engine compare: compare execution failure (python exit " + + "$pyRc, rust exit $rsRc). No engine's result was substituted for the other's failure. " + + "Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts in $cmpDir") ` + -ErrorAction Continue + if (-not $rsLegal) { Write-Error -Message "own-check: raw Rust child status: $rsRc" -ErrorAction Continue } + exit 5 + } + + # D4.1 (a)/(b): agreement or divergence, on bytes and the exit code. + $diverged = @() + if ($pyRc -ne $rsRc) { $diverged += "exit ($pyRc vs $rsRc)" } + $pyOutH = (Get-FileHash -LiteralPath (Join-Path $cmpDir "python.out") -Algorithm SHA256).Hash + $rsOutH = (Get-FileHash -LiteralPath (Join-Path $cmpDir "rust.out") -Algorithm SHA256).Hash + $pyErrH = (Get-FileHash -LiteralPath (Join-Path $cmpDir "python.err") -Algorithm SHA256).Hash + $rsErrH = (Get-FileHash -LiteralPath (Join-Path $cmpDir "rust.err") -Algorithm SHA256).Hash + if ($pyOutH -ne $rsOutH) { $diverged += "stdout" } + if ($pyErrH -ne $rsErrH) { $diverged += "stderr" } + if ($diverged.Count -gt 0) { + Write-Error -Message ("own-check: --engine compare: engine divergence — the reference and the " + + "candidate disagree on " + ($diverged -join ", ") + ". Neither verdict is exposed as " + + "authoritative. Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts " + + "in $cmpDir") -ErrorAction Continue + # Keep the artifacts for reproduction rather than deleting them. + $keep = $true + exit 5 + } + + # Agreement: the externally observed result is the reference's. + Get-Content -LiteralPath (Join-Path $cmpDir "python.out") -Raw -ErrorAction SilentlyContinue | Write-Output + $rc = $pyRc + } + finally { + if (-not $keep) { + Remove-Item -LiteralPath $cmpDir -Recurse -Force -ErrorAction SilentlyContinue + } + } + } } finally { Remove-Item $facts.FullName -ErrorAction SilentlyContinue diff --git a/scripts/own-check.sh b/scripts/own-check.sh index 8f937e42..e4ede189 100755 --- a/scripts/own-check.sh +++ b/scripts/own-check.sh @@ -12,10 +12,20 @@ # # Usage: # scripts/own-check.sh [--format human|github|msbuild|sarif] [--severity error|warning] +# [--engine python|rust|compare] # [--fail-on-finding] [--legacy] [--stats] [--body-throw-edges] # [--emit-facts ] [--config ] [--root ] # [--] [more ...] # +# --engine selects the analysis engine (#262 Stage 1). Python is the DEFAULT and +# the reference; `rust` runs the Rust core (`own-cli ownir`) instead; `compare` +# runs both over one captured input and exposes the reference's result only when +# they agree byte for byte. `rust` and `compare` require the candidate binary's +# absolute path in OWEN_RUST_CORE — there is no discovery of any kind, so an +# unset or unusable OWEN_RUST_CORE is a configuration error (exit 2) and NEVER a +# silent fall back to Python. A Rust failure is never turned into a Python +# success in any mode. +# # --config reads the project's [weak-subscription].subscribe wrapper # names (P-035) and teaches the extractor to treat those calls as already-released # weak subscriptions. A malformed config is a hard error. @@ -42,6 +52,9 @@ set -euo pipefail root="" format="human" severity="error" +# D1: Python is the Stage-1 default on every launcher surface. This line is the +# one that decides it for this surface. +engine="python" fail_on_finding=0 legacy=0 stats=0 @@ -61,6 +74,9 @@ while [[ $# -gt 0 ]]; do --severity) [[ $# -ge 2 ]] || { echo "own-check: --severity requires a value" >&2; exit 2; } severity="$2"; shift 2 ;; + --engine) + [[ $# -ge 2 ]] || { echo "own-check: --engine requires a value" >&2; exit 2; } + engine="$2"; shift 2 ;; --emit-facts) [[ $# -ge 2 ]] || { echo "own-check: --emit-facts requires a value" >&2; exit 2; } emit_facts="$2"; shift 2 ;; @@ -85,6 +101,50 @@ if [[ ${#paths[@]} -eq 0 ]]; then paths=(".") fi +case "$engine" in + python|rust|compare) ;; + *) echo "own-check: unknown --engine '$engine' (choose: python, rust, compare)" >&2; exit 2 ;; +esac + +# D3/D3.1 — the Stage-1 Rust candidate locator, resolved BEFORE anything is +# extracted, for the same reason the launcher resolves its engine runtime first: +# no point extracting facts just to fail on stage 2. OWEN_RUST_CORE is the one +# ratified spelling; there is NO discovery (no PATH lookup, no rust/target +# probing, no "first binary found"), because discovery is how a stale binary +# silently stands in for the one under test. Every rejection below is a +# configuration error (exit 2) — never exit 3 (that is Python-specific), never +# exit 5 (that is an internal failure), and never a fall back to Python. +rust_core="" +if [[ "$engine" == "rust" || "$engine" == "compare" ]]; then + rust_core="${OWEN_RUST_CORE:-}" + problem="" + if [[ -z "$rust_core" ]]; then + problem="is not set (or is empty)" + elif [[ -d "$rust_core" ]]; then + problem="points at a directory, not a file: '$rust_core'" + elif [[ ! -f "$rust_core" ]]; then + problem="points at a path that does not exist: '$rust_core'" + elif [[ ! -x "$rust_core" ]]; then + problem="points at a file that is not executable: '$rust_core'" + fi + if [[ -n "$problem" ]]; then + echo "own-check: --engine $engine needs the candidate \`own-cli\` binary, but OWEN_RUST_CORE $problem." >&2 + echo "own-check: set OWEN_RUST_CORE to the absolute path of the \`own-cli\` executable to run. Owen did not fall back to Python." >&2 + exit 2 + fi +fi + +# One digest helper for the compare evidence below: coreutils on Linux/git-bash, +# shasum where only that exists. Named once so the two sides cannot drift into +# two different hashes. +own_sha256() { + if command -v sha256sum >/dev/null 2>&1; then + sha256sum < "$1" | cut -d' ' -f1 + else + shasum -a 256 < "$1" | cut -d' ' -f1 + fi +} + extractor="$root/frontend/roslyn/OwnSharp.Extractor" facts="$(mktemp)" trap 'rm -f "$facts"' EXIT @@ -103,6 +163,11 @@ extractor_args=("${paths[@]}" -o "$facts") # weak-subscribe wrapper names ([weak-subscription].subscribe) and forward each to # the extractor as an internal transport flag. The Python carrier is the one place # that parses/validates the config; a malformed config is a hard error here. +# +# This carrier stays Python under EVERY --engine, including `rust`: it is a +# separately documented NON-CORE Python duty, not part of the core analysis +# seam, and it is #262's D9 tail (still an open owner decision). Making +# `--engine rust` skip or reimplement it here would be inventing D9's answer. if [[ -n "$config" ]]; then if ! weak_pairs="$(PYTHONPATH="$root" python -m ownlang config "$config")"; then exit 2 @@ -133,11 +198,125 @@ if [[ -n "$emit_facts" ]]; then cp "$facts" "$emit_facts" fi -# Stage 2: the one checker produces the verdict at the C# location. -set +e -PYTHONPATH="$root" python -m ownlang ownir "$facts" --format "$format" --severity "$severity" -rc=$? -set -e +# Stage 2: the selected engine produces the verdict at the C# location. +case "$engine" in + python) + set +e + PYTHONPATH="$root" python -m ownlang ownir "$facts" --format "$format" --severity "$severity" + rc=$? + set -e + ;; + + rust) + # The PRODUCTION Rust executable, never own-shadow-engine (that is #260's + # dev oracle and is not wired into production by this stage). Same argument + # vector as the reference: only the engine differs. + set +e + "$rust_core" ownir "$facts" --format "$format" --severity "$severity" + rc=$? + set -e + # 0/1/2 are verdicts and pass through. Anything else — 70, a panic, a + # signal death, an arbitrary 42 — is NOT a verdict: it takes the public + # internal-error path (5) with the raw status named on stderr, and it never + # runs Python instead. + if [[ "$rc" -ne 0 && "$rc" -ne 1 && "$rc" -ne 2 ]]; then + echo "own-check: the Rust analysis core exited $rc, which is not a verdict (raw child status: $rc). Owen did not fall back to Python." >&2 + exit 5 + fi + ;; + + compare) + # D4/D4.1 — both engines over ONE capture. + # + # The extractor already ran exactly once above; these bytes are read once + # into one file and each engine's input is materialised from THAT file and + # then re-hashed against it. "Both were handed the same path" is an + # assumption; two recorded digests equal to the capture's is a measurement, + # and it is the measurement that makes "compare fed the engines different + # bytes and still reported agreement" a control that can go red. + cmp_dir="$(mktemp -d)" + trap 'rm -f "$facts"; rm -rf "$cmp_dir"' EXIT + capture="$cmp_dir/capture.json" + cp "$facts" "$capture" + capture_sha="$(own_sha256 "$capture")" + + # A compare that judged nothing agrees about nothing: a zero-document run + # is a failure, not an agreement (a zero denominator wearing a pass). + if ! PYTHONPATH="$root" python - "$capture" <<'ZERODOC' +import json, sys +try: + doc = json.load(open(sys.argv[1], encoding="utf-8")) +except Exception: + # A document the strict door should refuse is a legitimate compare case + # (both engines must refuse it identically), so it is not "zero document". + sys.exit(0) +if not isinstance(doc, dict): + sys.exit(0) +units = ("components", "functions", "services", "effects", "protocols", "protocol_functions") +sys.exit(0 if any(isinstance(doc.get(k), list) and doc.get(k) for k in units) else 1) +ZERODOC + then + echo "own-check: --engine compare: the captured OwnIR contains nothing to analyse — a compare over zero documents proves nothing and is a failure, not an agreement." >&2 + exit 5 + fi + + py_in="$cmp_dir/python-input.json" + rs_in="$cmp_dir/rust-input.json" + cp "$capture" "$py_in" + cp "$capture" "$rs_in" + py_in_sha="$(own_sha256 "$py_in")" + rs_in_sha="$(own_sha256 "$rs_in")" + if [[ "$py_in_sha" != "$capture_sha" || "$rs_in_sha" != "$capture_sha" ]]; then + echo "own-check: --engine compare: the two engine inputs are not byte-identical to the single capture (capture $capture_sha, python $py_in_sha, rust $rs_in_sha) — the same-input invariant failed, so no comparison may be reported." >&2 + exit 5 + fi + + set +e + PYTHONPATH="$root" python -m ownlang ownir "$py_in" --format "$format" --severity "$severity" \ + >"$cmp_dir/python.out" 2>"$cmp_dir/python.err" + py_rc=$? + "$rust_core" ownir "$rs_in" --format "$format" --severity "$severity" \ + >"$cmp_dir/rust.out" 2>"$cmp_dir/rust.err" + rs_rc=$? + set -e + + # D4.1 (c): either engine failing to produce a verdict is an EXECUTION + # failure — checked before divergence, because two results are only + # comparable once both exist. No engine's answer substitutes for the + # other's failure. + py_legal=0; rs_legal=0 + [[ "$py_rc" -eq 0 || "$py_rc" -eq 1 || "$py_rc" -eq 2 ]] && py_legal=1 + [[ "$rs_rc" -eq 0 || "$rs_rc" -eq 1 || "$rs_rc" -eq 2 ]] && rs_legal=1 + if [[ "$py_legal" -eq 0 || "$rs_legal" -eq 0 ]]; then + echo "own-check: --engine compare: compare execution failure (python exit $py_rc, rust exit $rs_rc). No engine's result was substituted for the other's failure." >&2 + [[ "$rs_legal" -eq 0 ]] && echo "own-check: raw Rust child status: $rs_rc" >&2 + echo "own-check: reproduction — input sha256 $capture_sha, candidate $rust_core" >&2 + cat "$cmp_dir/python.err" >&2 || true + cat "$cmp_dir/rust.err" >&2 || true + exit 5 + fi + + # D4.1 (a)/(b): agreement or divergence, on BYTES and the exit code. + diverged="" + [[ "$py_rc" -ne "$rs_rc" ]] && diverged="exit ($py_rc vs $rs_rc)" + cmp -s "$cmp_dir/python.out" "$cmp_dir/rust.out" || diverged="${diverged:+$diverged, }stdout" + cmp -s "$cmp_dir/python.err" "$cmp_dir/rust.err" || diverged="${diverged:+$diverged, }stderr" + if [[ -n "$diverged" ]]; then + echo "own-check: --engine compare: engine divergence — the reference and the candidate disagree on $diverged. Neither verdict is exposed as authoritative." >&2 + echo "own-check: reproduction — input sha256 $capture_sha, candidate $rust_core, artifacts in $cmp_dir" >&2 + diff <(cat "$cmp_dir/python.out") <(cat "$cmp_dir/rust.out") >&2 || true + diff <(cat "$cmp_dir/python.err") <(cat "$cmp_dir/rust.err") >&2 || true + # Keep the artifacts for reproduction rather than deleting them on exit. + trap 'rm -f "$facts"' EXIT + exit 5 + fi + + # Agreement: the externally observed result is the Python/reference one. + cat "$cmp_dir/python.out" + cat "$cmp_dir/python.err" >&2 + rc=$py_rc + ;; +esac # rc: 0 = clean, 1 = findings, >=2 = a hard error (bad facts / drifted contract). if [[ "$fail_on_finding" -eq 1 ]]; then diff --git a/tests/stage1_campaign_layer.sh b/tests/stage1_campaign_layer.sh new file mode 100755 index 00000000..5933bff8 --- /dev/null +++ b/tests/stage1_campaign_layer.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# +# The Stage-1 campaign layer: build the launcher the mutation just edited, then +# run the engine controls against it. +# +# A mutation campaign edits PRODUCTION source and asks whether the controls +# notice. For the Rust and Python layers elsewhere in this repository the test +# runner compiles the mutated source itself, so a layer is one command. The +# Stage-1 launcher is C#: `tests/test_stage1_engine.py` drives an already-built +# `ownsharp.dll`, so a mutation to CheckCommand.cs or CompareMode.cs would be +# invisible to it unless the binary is rebuilt first. This script is that +# "first" — it exists so a mutated launcher is the launcher under test, rather +# than yesterday's build wearing today's source. +# +# A build failure is a real outcome and exits non-zero: the campaign records it +# as a catch whose name says the layer failed, which is honest — a mutation +# that does not compile produced no evidence either way, and pretending it was +# "caught by a test" would inflate the campaign. +# +# Usage (from the repository root, as the campaign invokes it): +# bash tests/stage1_campaign_layer.sh + +set -uo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$root" + +cli_proj="frontend/roslyn/OwnSharp.Cli/OwnSharp.Cli.csproj" +out="frontend/roslyn/OwnSharp.Cli/bin/Release/net8.0" + +# Rebuild the launcher from the (possibly mutated) source. +if ! dotnet build "$cli_proj" -c Release --nologo -v q; then + echo "stage1-layer: the launcher did not build" >&2 + exit 1 +fi + +# The vendored Python core is a PACK-time payload, so a plain build does not +# place it where the launcher looks. Stage it here for the Python-engine paths; +# this is harness setup, not a production behaviour. +mkdir -p "$out/ownlang-core/ownlang" +cp ownlang/*.py "$out/ownlang-core/ownlang/" + +export OWEN_STAGE1_LAUNCHER_DLL="$root/$out/ownsharp.dll" +# Every control must actually run inside a campaign: a skipped control cannot +# catch a mutation, and a campaign whose denominator quietly shrank is exactly +# the zero-denominator "green" the P-022 discipline refuses. +export OWEN_STAGE1_REQUIRE=1 + +exec python3 tests/test_stage1_engine.py diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py new file mode 100644 index 00000000..3648ad67 --- /dev/null +++ b/tests/test_stage1_engine.py @@ -0,0 +1,705 @@ +#!/usr/bin/env python3 +"""#262 Stage 1 — the launcher's engine-selection contract, measured. + +Stage 1 makes the Rust core *selectable* by the launcher while Python stays the +default and the reference. That is a claim about a seam, and a seam is proved +by driving it, not by reading it: every check below runs a real launcher +against a real candidate binary and asserts on the observable result — the exit +code, the streams, the evidence file. + +The fifteen ratified adversarial controls, each named by the misreading it +catches: + + default-stays-python the default silently becomes Rust + rust-actually-runs-rust explicit Rust selection actually runs Python + rust-failure-no-fallback a Rust failure runs Python + unexpected-rc-maps-to-5 an unexpected rc escapes as itself instead of 5 + raw-rc-retained the raw rc is lost from the evidence + rc70-is-not-a-verdict rc 70 is read as a finding or as clean + no-selector-in-own-cli engine selection leaks into `own-cli` + compare-extracts-once compare extracts twice + compare-same-input compare feeds the engines different bytes and + still reports agreement + compare-no-substitution one compare engine fails and the other's answer + is used + compare-zero-document a zero-document compare passes + candidate-identity a stale binary stands in without its recorded + identity changing + divergence-is-5 a divergence is exposed as a normal result, or as + exit 1, instead of 5 + exec-failure-is-5 a compare execution failure exits anything but 5, + or omits failure evidence + bad-locator-is-2 an invalid OWEN_RUST_CORE produces anything other + than exit 2, falls back to Python, or is mapped + to 3 or 5 + +Failures print `FAIL[]: ` so a mutation campaign names the CHECK +that caught it rather than whichever case tripped first, and the run never +stops at the first failure — a campaign needs every catcher a mutation trips, +not the earliest one. + +Forcing the failures uses #261's own off-by-default `fault-injection` feature +(`OWN_CLI_FAULT_PANIC`, `OWN_CLI_FAULT_ABORT`) against the REAL production +binary rather than a stub that only resembles one: a control that proves a +mock's behaviour proves nothing about the candidate. A stub is used for exactly +one case — pinning an arbitrary exit code such as 42, which no fault the real +binary offers produces on purpose. + +Toolchain. These controls need a built `own-cli` and a built launcher; the +environment names them: + + OWEN_RUST_CORE the production candidate (required) + OWEN_STAGE1_RUST_FAULT an `own-cli` built --features fault-injection + OWEN_STAGE1_LAUNCHER_DLL the built `ownsharp.dll` (the `owen` launcher) + OWEN_STAGE1_REQUIRE=1 every toolchain-dependent control MUST run + +`OWEN_STAGE1_REQUIRE=1` is the zero-denominator guard: without it a machine +lacking the toolchain skips and says so, but in CI — where the job exists to +provide that toolchain — a skip is indistinguishable from a pass, so the flag +turns every skip into a failure. + +Run: python tests/test_stage1_engine.py + python tests/run_tests.py (in the suite) +""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +SAMPLE_CS = """using System; +using System.IO; + +public class Leaky +{ + public void Run() + { + var s = new FileStream("x.txt", FileMode.OpenOrCreate); + Console.WriteLine(s.Length); + } +} +""" + +# A syntactically valid C# file with nothing an analysis can be about: it +# produces an OwnIR document whose collections are all empty, which is the +# zero-document case compare must refuse rather than call agreement. +EMPTY_CS = """namespace Nothing +{ +} +""" + +_FAILURES: list[tuple[str, str]] = [] +_PASSES: list[str] = [] +_SKIPS: list[tuple[str, str]] = [] + + +def fail(check: str, detail: str) -> None: + _FAILURES.append((check, detail)) + print(f"FAIL[{check}]: {detail}") + + +def ok(check: str, detail: str = "") -> None: + _PASSES.append(check) + print(f"ok[{check}]{': ' + detail if detail else ''}") + + +def skip(check: str, why: str) -> None: + """A skip is a real outcome, not a quiet pass — and under + OWEN_STAGE1_REQUIRE it is a failure, because the job that sets that flag + exists precisely to make the control runnable.""" + if os.environ.get("OWEN_STAGE1_REQUIRE") == "1": + fail(check, f"required control could not run: {why}") + return + _SKIPS.append((check, why)) + print(f"skip[{check}]: {why}") + + +# --- toolchain ------------------------------------------------------------- + + +def rust_core() -> str | None: + p = os.environ.get("OWEN_RUST_CORE") + return p if p and Path(p).is_file() else None + + +def rust_fault_core() -> str | None: + p = os.environ.get("OWEN_STAGE1_RUST_FAULT") + return p if p and Path(p).is_file() else None + + +def launcher_dll() -> str | None: + p = os.environ.get("OWEN_STAGE1_LAUNCHER_DLL") + if p and Path(p).is_file(): + return p + for cfg in ("Release", "Debug"): + cand = ROOT / "frontend/roslyn/OwnSharp.Cli/bin" / cfg / "net8.0/ownsharp.dll" + if cand.is_file(): + return str(cand) + return None + + +def have_dotnet() -> bool: + return shutil.which("dotnet") is not None + + +def run_own_check(args: list[str], env: dict[str, str] | None = None, + cwd: str | None = None) -> subprocess.CompletedProcess[bytes]: + """own-check.sh, with raw bytes: the compare contract is about bytes.""" + e = dict(os.environ) + e.update(env or {}) + return subprocess.run( + ["bash", str(ROOT / "scripts/own-check.sh"), *args], + capture_output=True, env=e, cwd=cwd or str(ROOT), check=False) + + +def run_owen(args: list[str], env: dict[str, str] | None = None + ) -> subprocess.CompletedProcess[bytes] | None: + dll = launcher_dll() + if dll is None or not have_dotnet(): + return None + e = dict(os.environ) + e.update(env or {}) + return subprocess.run( + ["dotnet", dll, "check", *args], + capture_output=True, env=e, cwd=str(ROOT), check=False) + + +def write_stub(path: Path, exit_code: int, stdout: str = "", stderr: str = "") -> Path: + """A candidate that exits with a chosen code. Unix only — the real + fault-injection binary covers both platforms for the cases it can force.""" + path.write_text( + "#!/usr/bin/env bash\n" + f"printf '%s' {json.dumps(stdout)}\n" + f"printf '%s' {json.dumps(stderr)} >&2\n" + f"exit {exit_code}\n", + encoding="utf-8") + path.chmod(0o755) + return path + + +# --- the controls ---------------------------------------------------------- + + +def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: + """D3.1: a missing/empty/nonexistent/non-file/non-executable + OWEN_RUST_CORE, for --engine rust|compare, is exit 2 — never 3 (that is + Python-specific), never 5 (that is an internal failure), and never a + fallback that quietly produces a Python answer.""" + check = "bad-locator-is-2" + not_exec = tmp / "not-executable" + not_exec.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + not_exec.chmod(0o644) + a_dir = tmp / "a-directory" + a_dir.mkdir(exist_ok=True) + + cases = { + "unset": None, + "empty": "", + "nonexistent": str(tmp / "definitely-absent"), + "directory": str(a_dir), + "non-executable": str(not_exec), + } + problems = [] + for engine in ("rust", "compare"): + for name, value in cases.items(): + env = {"OWEN_RUST_CORE": value} if value is not None else {} + e = dict(os.environ) + e.update(env) + if value is None: + e.pop("OWEN_RUST_CORE", None) + r = subprocess.run( + ["bash", str(ROOT / "scripts/own-check.sh"), + "--engine", engine, "--", str(sample)], + capture_output=True, env=e, cwd=str(ROOT), check=False) + if r.returncode != 2: + problems.append(f"{engine}/{name}: exit {r.returncode}, expected 2") + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + # A fallback would have produced a verdict; the diagnostic must + # also say, in as many words, that no fallback happened. + if "finding" in merged and "OWEN_RUST_CORE" not in merged: + problems.append(f"{engine}/{name}: produced a verdict — looks like a fallback") + if r.returncode == 2 and "did not fall back to Python" not in merged: + problems.append(f"{engine}/{name}: diagnostic does not deny a Python fallback") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, f"{len(cases) * 2} invalid-locator cases all exit 2, no fallback") + + +def control_default_stays_python(sample: Path) -> None: + """D1: the default engine is Python. Proved NEGATIVELY and positively: a + default run with a deliberately unusable Python must fail on Python (the + launcher's exit 3), which it cannot do if the default silently moved to + Rust; and it must not produce a Rust-only success.""" + check = "default-stays-python" + r = run_owen(["--format", "human", str(sample)], + env={"OWEN_PYTHON": "/definitely/not/a/python"}) + if r is None: + skip(check, "no built launcher/dotnet") + return + if r.returncode != 3: + fail(check, f"default run with a broken OWEN_PYTHON exited {r.returncode}, expected 3 " + "(the default engine is not Python any more, or Python is no longer resolved " + "for it)") + return + ok(check, "the default still resolves Python and fails on it (exit 3)") + + +def control_rust_actually_runs_rust(sample: Path) -> None: + """Explicit Rust selection must actually run Rust. Proved by breaking + Python so thoroughly that a Python run could not succeed, and requiring the + Rust run to succeed anyway — which also proves the launcher did not resolve + Python or unpack the vendored core for a Rust-only invocation.""" + check = "rust-actually-runs-rust" + core = rust_core() + if core is None: + skip(check, "no OWEN_RUST_CORE") + return + r = run_owen(["--engine", "rust", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": core, "OWEN_PYTHON": "/definitely/not/a/python"}) + if r is None: + skip(check, "no built launcher/dotnet") + return + if r.returncode == 3: + fail(check, "explicit --engine rust exited 3 (no usable Python) — the Rust path " + "still resolves Python") + return + if r.returncode not in (0, 1): + fail(check, f"explicit --engine rust exited {r.returncode} with a broken OWEN_PYTHON; " + f"stderr={r.stderr.decode('utf-8', 'replace')[:400]}") + return + if b"OWN001" not in r.stdout: + fail(check, "explicit --engine rust produced no finding for a known-leaky sample") + return + ok(check, "Rust ran and produced the verdict with Python unusable") + + +def control_rust_failure_no_fallback(sample: Path) -> None: + """A Rust failure is never a Python success. The candidate is forced to + fail; the launcher must not answer with Python's verdict.""" + check = "rust-failure-no-fallback" + fault = rust_fault_core() + if fault is None: + skip(check, "no OWEN_STAGE1_RUST_FAULT") + return + r = run_owen(["--engine", "rust", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": fault, "OWN_CLI_FAULT_PANIC": "1"}) + if r is None: + skip(check, "no built launcher/dotnet") + return + problems = [] + if r.returncode in (0, 1): + problems.append(f"owen: a forced Rust failure exited {r.returncode} — a verdict was " + "produced despite the engine failing (a fallback, or the failure was " + "swallowed)") + elif r.returncode != 5: + problems.append(f"owen: a forced Rust failure exited {r.returncode}, expected public 5") + if b"OWN001" in r.stdout: + problems.append("owen: a forced Rust failure still produced findings — Python answered " + "for Rust") + + # The same contract on the shell surface (D2): one contract, both surfaces. + r2 = run_own_check(["--engine", "rust", "--format", "human", "--", str(sample)], + env={"OWEN_RUST_CORE": fault, "OWN_CLI_FAULT_PANIC": "1"}) + if r2.returncode in (0, 1): + problems.append(f"own-check.sh: a forced Rust failure exited {r2.returncode} — a verdict " + "was produced despite the engine failing") + if b"OWN001" in r2.stdout: + problems.append("own-check.sh: a forced Rust failure still produced findings — Python " + "answered for Rust") + + if problems: + fail(check, "; ".join(problems)) + return + ok(check, "a forced Rust failure produces no verdict on either launcher surface") + + +def control_rc70_is_not_a_verdict(sample: Path) -> None: + """rc 70 is the engines' shared internal-error code: known, still a + failure. Reading it as findings (1) or as clean (0) is the bug.""" + check = "rc70-is-not-a-verdict" + fault = rust_fault_core() + if fault is None: + skip(check, "no OWEN_STAGE1_RUST_FAULT") + return + # The forced panic is #261's measured rc-70 path. + raw = subprocess.run([fault, "ownir", "--format", "human", "/nonexistent-facts.json"], + capture_output=True, check=False, + env={**os.environ, "OWN_CLI_FAULT_PANIC": "1"}) + if raw.returncode != 70: + skip(check, f"the fault build did not produce rc 70 (got {raw.returncode})") + return + r = run_owen(["--engine", "rust", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": fault, "OWN_CLI_FAULT_PANIC": "1"}) + if r is None: + skip(check, "no built launcher/dotnet") + return + if r.returncode in (0, 1): + fail(check, f"a Rust child rc 70 surfaced as {r.returncode} — read as " + f"{'clean' if r.returncode == 0 else 'findings'} instead of a failure") + return + if r.returncode != 5: + # Anything else means 70 was treated as a legal engine result and + # passed through, rather than taking Owen's internal-error path. + fail(check, f"a Rust child rc 70 surfaced as {r.returncode}, expected public 5 " + "(70 is a known failure, never a verdict)") + return + ok(check, "a Rust child rc 70 takes the public internal-error path (exit 5)") + + +def control_unexpected_rc_and_raw_retention(sample: Path, tmp: Path) -> None: + """An unexpected child status maps to public 5, and the RAW status is + retained in the diagnostic report's typed `child_exit_code` (D5). Two + controls, one forced condition: the mapping and the retention fail + independently and are reported independently.""" + map_check, keep_check = "unexpected-rc-maps-to-5", "raw-rc-retained" + if os.name == "nt": + skip(map_check, "stub candidate is Unix-only") + skip(keep_check, "stub candidate is Unix-only") + return + stub = write_stub(tmp / "rc42-core", 42, stderr="forced unexpected status\n") + report = Path.home() / ".owen/diag/last-failure.json" + if report.exists(): + report.unlink() + r = run_owen(["--engine", "rust", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": str(stub)}) + if r is None: + skip(map_check, "no built launcher/dotnet") + skip(keep_check, "no built launcher/dotnet") + return + + if r.returncode == 42: + fail(map_check, "an unexpected child rc 42 escaped as the public exit 42") + elif r.returncode != 5: + fail(map_check, f"an unexpected child rc 42 surfaced as {r.returncode}, expected public 5") + else: + ok(map_check, "an unexpected child rc 42 maps to public exit 5") + + if not report.exists(): + fail(keep_check, "no diagnostic report was written for an unexpected child status") + return + try: + data = json.loads(report.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + fail(keep_check, f"the diagnostic report is unreadable: {exc}") + return + if "child_exit_code" not in data: + fail(keep_check, "the report has no `child_exit_code` field (D5)") + return + if data["child_exit_code"] != 42: + fail(keep_check, f"`child_exit_code` is {data['child_exit_code']!r}, expected the raw 42") + return + if not isinstance(data["child_exit_code"], int): + fail(keep_check, "`child_exit_code` is not a typed integer") + return + if data.get("schema") != 2: + fail(keep_check, f"the report schema is {data.get('schema')!r}, expected 2 (D5 bumped it)") + return + ok(keep_check, "the raw 42 is retained as a typed child_exit_code, schema 2") + + +def control_no_selector_in_own_cli() -> None: + """C-4: `own-cli` presents ONE engine and knows nothing of Python. Engine + selection must not have leaked into it — neither as an accepted flag nor as + a source-level concept. + + Note what is deliberately NOT asserted: that the word "python" is absent + from `own-cli`'s output. Its usage text is the REFERENCE's module + docstring, frozen byte for byte as #261's C-1 measured it, and that + docstring says `python -m ownlang ...` because that is what the reference + prints. Failing on that string would be a control that fires on the + contract being kept. + """ + check = "no-selector-in-own-cli" + core = rust_core() + if core is None: + skip(check, "no OWEN_RUST_CORE") + return + problems = [] + + # An engine selector must not be ACCEPTED. `own-cli` treats it as an + # unknown argument (the reference's own behaviour: a second positional), + # which is a usage error — never a selection. + r = subprocess.run([core, "ownir", "--engine", "python", "/nonexistent.json"], + capture_output=True, check=False) + if r.returncode == 0: + problems.append("`own-cli ownir --engine python` succeeded — it accepts a selector") + + # And it must not be a source-level concept: no engine environment + # variable, no selection branch. Comments explaining what `own-cli` is NOT + # are exactly where this boundary is documented, so they are excluded. + crate = ROOT / "rust/crates/own-cli/src" + for path in sorted(crate.rglob("*.rs")): + for line in path.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if stripped.startswith(("//", "//!", "///", "*", "#!")): + continue + for needle in ("OWEN_RUST_CORE", "OWEN_PYTHON", "OWN_PYTHON"): + if needle in stripped: + problems.append(f"{path.name} reads {needle!r} in code: {stripped[:80]}") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "own-cli accepts no engine selector and reads no engine environment") + + +def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None: + """D4/#260: compare extracts ONCE and hands both engines byte-identical + input derived from that one capture. + + Extraction count is measured behaviourally, with a `dotnet` shim first on + PATH that records every invocation: the extractor is driven through + `dotnet run`, so counting those invocations counts extractions. Same-input + is measured by the launcher's own refusal to report a comparison whose two + engine inputs do not hash to the capture. + """ + once_check, same_check = "compare-extracts-once", "compare-same-input" + core = rust_core() + if core is None or not have_dotnet(): + skip(once_check, "no OWEN_RUST_CORE/dotnet") + skip(same_check, "no OWEN_RUST_CORE/dotnet") + return + + shim_dir = tmp / "shim" + shim_dir.mkdir(exist_ok=True) + tally = tmp / "dotnet-invocations.log" + if tally.exists(): + tally.unlink() + real_dotnet = shutil.which("dotnet") + (shim_dir / "dotnet").write_text( + "#!/usr/bin/env bash\n" + f'printf "%s\\n" "$*" >> {json.dumps(str(tally))}\n' + f'exec {json.dumps(str(real_dotnet))} "$@"\n', + encoding="utf-8") + (shim_dir / "dotnet").chmod(0o755) + + env = {"OWEN_RUST_CORE": core, "PATH": f"{shim_dir}{os.pathsep}{os.environ.get('PATH','')}"} + r = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], env=env) + if r.returncode not in (0, 1): + fail(once_check, f"compare over a healthy sample exited {r.returncode}: " + f"{r.stderr.decode('utf-8', 'replace')[-400:]}") + fail(same_check, "compare did not reach agreement, so same-input could not be observed") + return + + lines = tally.read_text(encoding="utf-8").splitlines() if tally.exists() else [] + extractions = [ln for ln in lines if "OwnSharp.Extractor" in ln] + if len(extractions) != 1: + fail(once_check, f"compare invoked the extractor {len(extractions)} times, expected exactly 1") + else: + ok(once_check, "compare extracted exactly once") + + # Same-input, MEASURED at the candidate rather than inferred from the + # launcher's own bookkeeping: a stub candidate records the sha256 of the + # file it was actually handed, and that digest must equal the capture + # digest the launcher attests in its evidence. If compare ever fed the two + # engines different bytes, these two values part company. + if os.name == "nt": + skip(same_check, "recording stub is Unix-only") + return + seen = tmp / "candidate-saw.sha256" + recorder = tmp / "recording-core" + recorder.write_text( + "#!/usr/bin/env bash\n" + "# args: ownir --format F --severity S\n" + f"sha256sum < \"$2\" | cut -d' ' -f1 > {json.dumps(str(seen))}\n" + "exit 0\n", + encoding="utf-8") + recorder.chmod(0o755) + + evidence = Path.home() / ".owen/compare/last-compare.json" + if evidence.exists(): + evidence.unlink() + r2 = run_owen(["--engine", "compare", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": str(recorder)}) + if r2 is None: + skip(same_check, "no built launcher/dotnet") + return + if not seen.exists(): + fail(same_check, "the candidate was never handed an input file to record") + return + if not evidence.exists(): + fail(same_check, "the compare run wrote no evidence to attest the capture digest") + return + try: + attested = (json.loads(evidence.read_text(encoding="utf-8")).get("input") or {}).get("sha256") + except (OSError, json.JSONDecodeError) as exc: + fail(same_check, f"the compare evidence is unreadable: {exc}") + return + candidate_saw = seen.read_text(encoding="utf-8").strip() + if candidate_saw != attested: + fail(same_check, f"the candidate was handed bytes hashing to {candidate_saw}, but the " + f"launcher attested the capture as {attested} — the engines did not " + "receive the same input") + return + ok(same_check, f"the candidate received exactly the attested capture ({candidate_saw[:16]}…)") + + +def control_compare_zero_document(tmp: Path) -> None: + """A compare that judged nothing agrees about nothing: a zero-document run + is a failure, not an agreement — a zero denominator wearing a pass.""" + check = "compare-zero-document" + core = rust_core() + if core is None or not have_dotnet(): + skip(check, "no OWEN_RUST_CORE/dotnet") + return + empty_dir = tmp / "empty-sample" + empty_dir.mkdir(exist_ok=True) + (empty_dir / "Nothing.cs").write_text(EMPTY_CS, encoding="utf-8") + r = run_own_check(["--engine", "compare", "--format", "human", "--", str(empty_dir)], + env={"OWEN_RUST_CORE": core}) + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode in (0, 1): + fail(check, f"a zero-document compare exited {r.returncode} — it passed instead of failing") + return + if r.returncode != 5: + # Exit 4 (no supported input) is a different, legitimate refusal: the + # sample never reached the engines at all, so the control cannot speak. + if r.returncode == 4: + skip(check, "the sample was rejected as unsupported input before the engines ran") + return + fail(check, f"a zero-document compare exited {r.returncode}, expected 5") + return + if "nothing to analyse" not in merged and "zero document" not in merged: + fail(check, "a zero-document compare failed without saying why") + return + ok(check, "a zero-document compare fails (exit 5) and says so") + + +def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: + """D4.1 (b)/(c): a divergence and an execution failure both exit 5 with + evidence, and neither ever substitutes one engine's answer for the + other's.""" + div_check = "divergence-is-5" + exec_check = "exec-failure-is-5" + sub_check = "compare-no-substitution" + if os.name == "nt": + for c in (div_check, exec_check, sub_check): + skip(c, "stub candidate is Unix-only") + return + if not have_dotnet(): + for c in (div_check, exec_check, sub_check): + skip(c, "no dotnet") + return + + # (b) divergence: a candidate that answers legally but differently. + diverging = write_stub(tmp / "diverging-core", 0, stdout="a different answer\n") + r = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], + env={"OWEN_RUST_CORE": str(diverging)}) + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode == 1: + fail(div_check, "a compare divergence exited 1 — in public Owen that already means findings") + elif r.returncode != 5: + fail(div_check, f"a compare divergence exited {r.returncode}, expected public 5") + elif "divergence" not in merged: + fail(div_check, "a compare divergence exited 5 without an actionable diagnostic") + elif "sha256" not in merged: + fail(div_check, "a compare divergence produced no reproduction evidence (no input digest)") + else: + ok(div_check, "a compare divergence is public exit 5 with reproduction evidence") + + if b"a different answer" in r.stdout: + fail(sub_check, "the candidate's answer was exposed as the result of a diverging compare") + else: + ok(sub_check, "no engine's answer was exposed on divergence") + + # (c) execution failure: a candidate that produces no verdict at all. + crashing = write_stub(tmp / "crashing-core", 42, stderr="forced execution failure\n") + r2 = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], + env={"OWEN_RUST_CORE": str(crashing)}) + merged2 = (r2.stdout + r2.stderr).decode("utf-8", "replace") + if r2.returncode != 5: + fail(exec_check, f"a compare execution failure exited {r2.returncode}, expected public 5") + elif "execution failure" not in merged2: + fail(exec_check, "a compare execution failure exited 5 without an actionable diagnostic") + elif "42" not in merged2: + fail(exec_check, "a compare execution failure did not retain the raw Rust child status") + else: + ok(exec_check, "a compare execution failure is public exit 5 with failure evidence") + + if b"OWN001" in r2.stdout: + fail(sub_check, "Python's verdict was exposed after the candidate failed the compare") + + +def control_candidate_identity(sample: Path, tmp: Path) -> None: + """D3: the candidate's identity is recorded, so a wrong or stale binary + cannot stand in without the evidence changing.""" + check = "candidate-identity" + core = rust_core() + if core is None or not have_dotnet(): + skip(check, "no OWEN_RUST_CORE/dotnet") + return + evidence = Path.home() / ".owen/compare/last-compare.json" + if evidence.exists(): + evidence.unlink() + r = run_owen(["--engine", "compare", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": core}) + if r is None: + skip(check, "no built launcher/dotnet") + return + if not evidence.exists(): + fail(check, "a compare run wrote no evidence file") + return + try: + data = json.loads(evidence.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + fail(check, f"the compare evidence is unreadable: {exc}") + return + recorded = (data.get("rust_core") or {}).get("sha256") + actual = hashlib.sha256(Path(core).read_bytes()).hexdigest() + if recorded != actual: + fail(check, f"the evidence records candidate sha256 {recorded!r}, the binary that ran " + f"hashes to {actual}") + return + if not (data.get("input") or {}).get("sha256"): + fail(check, "the compare evidence records no input digest — the same-input claim is " + "unattested") + return + if (data.get("rust_core") or {}).get("bytes") != Path(core).stat().st_size: + fail(check, "the evidence's recorded candidate byte length does not match the binary") + return + ok(check, "the compare evidence records the candidate's sha256 and byte length") + + +# --- harness --------------------------------------------------------------- + + +def run() -> int: + with tempfile.TemporaryDirectory(prefix="owen-stage1-") as td: + tmp = Path(td) + sample_dir = tmp / "sample" + sample_dir.mkdir() + (sample_dir / "Leak.cs").write_text(SAMPLE_CS, encoding="utf-8") + + # No fail-fast: every control runs, so a campaign sees every catcher a + # mutation trips rather than only the first. + control_bad_locator_is_2(sample_dir, tmp) + control_no_selector_in_own_cli() + control_default_stays_python(sample_dir) + control_rust_actually_runs_rust(sample_dir) + control_rust_failure_no_fallback(sample_dir) + control_rc70_is_not_a_verdict(sample_dir) + control_unexpected_rc_and_raw_retention(sample_dir, tmp) + control_compare_same_input_and_extract_once(sample_dir, tmp) + control_compare_zero_document(tmp) + control_compare_failure_and_divergence(sample_dir, tmp) + control_candidate_identity(sample_dir, tmp) + + print() + print(f"stage-1 engine controls: {len(_PASSES)} passed, " + f"{len(_FAILURES)} failed, {len(_SKIPS)} skipped") + if _SKIPS: + print(" skipped (set OWEN_STAGE1_REQUIRE=1 to make these failures):") + for name, why in _SKIPS: + print(f" {name}: {why}") + return 1 if _FAILURES else 0 + + +if __name__ == "__main__": + sys.exit(run()) From 9f34e2058de42ec8464a72b0180bda011a541a08 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 01:30:04 +0000 Subject: [PATCH 02/22] docs(P-022): CI gate, generated-counts fragment and status surfaces for Stage 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI: a `stage1-engine` job on ubuntu AND windows. Both platforms matter here rather than being extra coverage — the ratified evidence is explicit Rust-selected runs on each, and the two differ in exactly the mechanics (process launch, executable bits, path forms) a Linux-only job leaves unproven. It builds the production own-cli and a second own-cli with #261's off-by-default fault-injection feature, so the failure-mode controls force failures through the real binary rather than a mock, and runs the controls under OWEN_STAGE1_REQUIRE=1 so a control that cannot run is a failure rather than a silently shrinking denominator. The Windows leg additionally drives own-check.ps1's engine contract, which no Linux leg can reach. Evidence: the Stage-1 campaign is rendered into docs/generated/p022-stage1-mutations.md by render_checkpoint_status.py, so its counts are generated from the recorded run and never typed, and tests/test_checkpoint_status.py fails while the fragment is stale. Status surfaces move together, describing what becomes true if this exact PR merges: P-022 row 8 and the preferred queue, the proposals index row, rust/README.md's own-cli row, and the launcher README (which gains the user-facing engine documentation and the explicit rollback). Each says plainly what Stage 1 is NOT: not the cutover, Python still default and reference, nothing public defaulting to Rust, Python distribution untouched, compare not yet a promised public feature. The Windows A/B/C behaviour change is named where a user meets it — compare will report a real divergence there for non-ASCII output, and that is the declared difference being visible. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .github/workflows/ci.yml | 111 ++++++++++++++++++++ docs/proposals/P-022-rust-core-migration.md | 4 +- docs/proposals/README.md | 2 +- frontend/roslyn/OwnSharp.Cli/README.md | 39 ++++++- rust/README.md | 2 +- scripts/render_checkpoint_status.py | 35 +++++- 6 files changed, 186 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ba9347a4..08d0276c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -108,6 +108,117 @@ jobs: - name: cargo test -p own-cli --features fault-injection (the failure-mode controls) run: cargo test -p own-cli --features fault-injection --test faults + # P-022 step 8 (#262) STAGE 1 — the launcher's engine-selection contract. + # + # Stage 1 makes the Rust core SELECTABLE while Python stays the default and + # the reference. Every claim in that sentence is a behaviour, so this job + # drives real launchers against a real candidate binary and asserts on the + # observable result. It is deliberately BOTH platforms: the ratified evidence + # is explicit Rust-selected runs on Windows and Linux, and the two differ in + # exactly the mechanics (process launch, executable bits, path forms) that a + # Linux-only job would leave unproven. + # + # OWEN_STAGE1_REQUIRE=1 is the zero-denominator guard: this job exists to + # provide the toolchain, so a control that skips here is a control that did + # not run, and a run of skips would otherwise be indistinguishable from a + # green one. + stage1-engine: + name: owen --engine (#262 Stage 1 controls) + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, windows-latest] + runs-on: ${{ matrix.os }} + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + - uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4 + with: + dotnet-version: "8.0.x" + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.13" + - uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10 + with: + toolchain: stable + # The PRODUCTION candidate. The controls run the real `own-cli`, not a + # stand-in: a control that proves a mock's behaviour proves nothing about + # the binary the launcher will actually spawn. + - name: Build the production own-cli candidate + working-directory: rust + run: cargo build -p own-cli --release + # A second build with #261's off-by-default fault-injection feature, so + # the failure-mode controls force failures through the real binary. + - name: Build the fault-injection own-cli (forced failure modes) + working-directory: rust + run: cargo build -p own-cli --release --features fault-injection --target-dir target-fault + - name: Build the owen launcher + run: dotnet build frontend/roslyn/OwnSharp.Cli/OwnSharp.Cli.csproj -c Release --nologo + # The vendored Python core is a PACK-time payload, so a plain build does + # not place it where the launcher looks. Staging it here is harness setup + # for the Python-engine paths, not a production behaviour. + - name: Stage the vendored core beside the built launcher + run: | + out=frontend/roslyn/OwnSharp.Cli/bin/Release/net8.0 + mkdir -p "$out/ownlang-core/ownlang" + cp ownlang/*.py "$out/ownlang-core/ownlang/" + - name: Stage-1 engine controls (all fifteen, no fail-fast) + env: + OWEN_STAGE1_REQUIRE: "1" + run: | + ext="" + if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi + export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli$ext" + export OWEN_STAGE1_RUST_FAULT="$PWD/rust/target-fault/release/own-cli$ext" + export OWEN_STAGE1_LAUNCHER_DLL="$PWD/frontend/roslyn/OwnSharp.Cli/bin/Release/net8.0/ownsharp.dll" + python tests/test_stage1_engine.py + # The explicit Rust-selected run on this platform, through the shell + # launcher, recorded as its own step so the evidence names the surface + # and the platform rather than being inferred from a green job. + - name: Explicit Rust-selected run through own-check.sh + run: | + ext="" + if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi + export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli$ext" + mkdir -p "$RUNNER_TEMP/stage1-sample" + cat > "$RUNNER_TEMP/stage1-sample/Leak.cs" <<'CS' + using System; + using System.IO; + public class Leaky + { + public void Run() + { + var s = new FileStream("x.txt", FileMode.OpenOrCreate); + Console.WriteLine(s.Length); + } + } + CS + set +e + out=$(bash scripts/own-check.sh --engine rust --format human --fail-on-finding -- "$RUNNER_TEMP/stage1-sample") + rc=$? + set -e + echo "$out" + [ "$rc" -eq 1 ] || { echo "FAIL: --engine rust expected exit 1 (findings), got $rc"; exit 1; } + case "$out" in *OWN001*) ;; *) echo "FAIL: no OWN001 finding from the Rust engine"; exit 1 ;; esac + echo "Rust-selected run OK on ${{ matrix.os }}" + # The Windows twin of the shell surface. own-check.ps1 is never exercised + # by the Linux legs, so its engine contract needs its own step here. + - name: own-check.ps1 -Engine rust (Windows only) + if: matrix.os == 'windows-latest' + shell: pwsh + run: | + $env:OWEN_RUST_CORE = "$PWD/rust/target/release/own-cli.exe" + $sample = Join-Path $env:RUNNER_TEMP "stage1-sample" + & ./scripts/own-check.ps1 -Engine rust -Format human -FailOnFinding -- $sample + if ($LASTEXITCODE -ne 1) { throw "own-check.ps1 -Engine rust expected exit 1, got $LASTEXITCODE" } + # And the D3.1 seam on this surface: an unusable locator is exit 2. + $env:OWEN_RUST_CORE = "C:\definitely\not\a\binary.exe" + & ./scripts/own-check.ps1 -Engine rust -Format human -- $sample + if ($LASTEXITCODE -ne 2) { throw "own-check.ps1 with a bad OWEN_RUST_CORE expected exit 2, got $LASTEXITCODE" } + Write-Host "own-check.ps1 engine contract OK" + # P-022 step 7a (#260) — COMPARE MODE over the committed corpus: the FAST half # of #260's test matrix, and one leg of it. The five pinned OSS repositories, # the large-solution controls and the examples tree are the scheduled/manual diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 430f475a..17733f03 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -76,9 +76,9 @@ was #258 alone, which is satisfied. Per the checkpoints #259 itself defines: | 6b | Rust `own-bridge`, layered OwnIR parity | #259 | **final acceptance reached** — see the checkpoint table and the line above it | | 7a | dual-engine shadow mode + zero-diff reproduction artifacts | #260 (supported by #269) | **final acceptance REACHED**. The only wording it earns: *dual-engine compare mode reports zero acceptance-unexplained over its full test matrix — the committed corpus, the C# samples, the examples, the five pinned OSS repositories of #243 and the large-solution controls — at all three layers and on the derived SARIF, on byte-attested same input, with the OD-1 typed-door boundaries declared by policy; Python remains the public engine.* It is **not** "P-022 done" and **not** "Rust is the default", which is #262's cutover behind #261. The sweep is what the acceptance surfaces over the committed corpus deliberately left owed: ten documents over six targets, each repository at its **verified** pin (drift is a failed target, never a newer measurement), each extracted **once** through `own-check.sh --emit-facts` and compared from those bytes — the five directory walks, the largest `.sln` of every target that has one (a different extractor path, and measurably a differently *ordered* document rather than a subset), and `examples/`. Coverage is defined so that it cannot be faked: a repository is not covered because extraction succeeded, so the driver fails a run that compared zero documents AND a declared target it never reached, and the **denominators are recorded per target**. The driver gained the identity the #342 review asked for — every result and failure report names the adapter by `sha256` and byte length, taken from the file that ran — plus manifest runs whose every document is verified against its `facts_sha256` before any engine starts (`shadow_compare_version` 2; the artifact format v3 is untouched). Taking the measurement found six HARNESS defects and no engine divergence: a cross-drive `relpath` that killed the driver on a label, a timeout that never returned when the adapter had children, a control group that could not execute on Windows at all (and so had never caught the timeout one), and three in the mutation harness that between them meant no campaign could be recorded anywhere but Linux — rewritten line endings that made it refuse its own run, a catcher name that took the host's path separator and so reported five protected rules as unprotected, and a layer decoded with the console codepage. The five repositories' facts documents are not committed — their identities are. The scheduled/manual gate is `.github/workflows/shadow-sweep.yml`; every count lives in the generated fragments ([sweep](../generated/p022-shadow-sweep.md), [census](../generated/p022-shadow-census.md), [campaigns](../generated/p022-shadow-mutations.md)) and never here; the records are [the sweep note](../notes/p022-shadow-sweep.md) and [the acceptance note](../notes/p022-shadow-acceptance.md), which name what is measured-not-claimed. The owner decisions remain D-4..D-7, B-2, B-3, R-1 and R-2 in [the ledger](../notes/p022-shadow-infra-owner-decisions.md), unreopened. No production behaviour changed | | 7b | Rust `own-cli`: the production OwnIR executable — command/output/exit-code parity behind the existing launcher | #261 (residual `.own`/dev CLI: #345) | **261.A ratified; 261.B built, repaired to the ratified acceptance, and replaying on both platforms; #261 closed completed 2026-09-08 (PR #347, `206e9c7`).** Owner decisions C-1..C-5 (2026-09-08, recorded verbatim in #261) are unchanged and were applied, not re-litigated. What exists now: the `own-cli` binary with its single `ownir` subcommand, a Python-authored CLI fixture family (`tests/fixtures/cli_ownir/`) replayed against the built binary with **zero Python** on Linux and Windows CI, and an off-by-default `fault-injection` feature under which both failure-mode rulings are MEASURED: a catchable panic is one actionable stderr diagnostic and exit 70 (never 101) via a hook plus a top-level `catch_unwind` under `panic = "unwind"`, and an uncatchable death is a visible hard failure with no OS exit number contracted. The top-level shell follows the public `owen` convention as a parity surface of its own, written once and shared between the binary and the fixture; everything after `ownir` is the reference's own behaviour as measured, the docstring-on-stdout class frozen AND flagged so the owner can declare it a defect knowing what was frozen. The renders are reused, never re-derived; what the CLI adds is the SARIF serialization the reference's `cmd_ownir` uses (`json.dumps(indent=2)`, ASCII-escaped), which is not the BR-V9 goldens' byte shape. DAG: `own-cli -> own-ir`/`own-bridge` and nothing else. 261.B is built and replaying on both platforms, with the four rulings settled as follows: (2a) the `ownir_version` Version messages are **byte-parity** — that text is ours on both sides, so the divergence was a Rust bug and was fixed rather than declared. A second repair pass re-took that measurement over value **classes** rather than four hand-picked values and found three more defects a single-key, integer-valued control could not reach (CPython's dict order, its float spelling, and the arbitrary-precision integer that changes which branch the reference takes), plus a round-half-to-even tie found by a 200 000-double sweep. Re-measured: 24/24 classes and 20 000 randomized documents byte-identical, with V1 (the reference's non-standard JSON constants), V2 (the literal `-0`) and V4 (the two sides' independently versioned Unicode tables — a representation-only boundary, the mismatch count being a specific two-version measurement recorded in the note, not a fixed size) declared and excluded rather than counted. The census is now a Rust test replayed with zero Python; (2b) the JSON parser detail is a **declared typed boundary, CLI-B1** — the CLI-owned wrapper `{path}: error: {path} is not valid JSON: ` is pinned byte-exact and only the parser library's own text after it is declared, guarded by an executable `kind == Json` proof and a negative control that runs ONE case — one argv, one exact path, one decode route — against two byte sequences, so eligibility can only turn on the facts bytes; a Json rejection that loses its internal prefix fails onto rc 70 rather than passing through; (1) invalid UTF-8 is a **declared defect of the Python reference**, excluded from the byte contract pending a Python-first hygiene tail (`UnicodeDecodeError` -> `OwnIRError` -> rc 2) to close before public cutover, recorded in #262 — the tracker of record — and mirrored in #250's Still missing list; (3) Windows is **A** canonical reference parity plus **B** Rust portability, with **C** native-Windows Python parity explicitly **NOT claimed** — the reference there emits cp1252/CRLF and can fail with `UnicodeEncodeError`, recorded in #262 — the tracker of record — as a behavior change rather than parity, and mirrored in #250's Still missing list. Every count is generated (`docs/generated/p022-cli-census.md`, `docs/generated/p022-cli-mutations.md`); the record is [the note](../notes/p022-cli-ownir.md). The executable lives **behind** the unchanged `owen` launcher: nothing is wired, published or defaulted — that is #262 | -| 8 | Rust-default **cutover**, rollback gate, Python distribution removal | #262 | its #261 prerequisite — the production OwnIR executable alone (C-5) — is satisfied: 261.B landed (PR #347, `206e9c7`) and #261 is closed completed; #260 reached; #345 is not on this path. #263's baselines are the evidence prerequisite of the cutover decision, not a normative blocker. Launcher rulings recorded in #262: engine selection is the launcher's, never the executable's; an unexpected Rust child exit code outside the legal set takes the public internal-error path with the raw child status retained in the evidence; no silent fallback | +| 8 | Rust-default **cutover**, rollback gate, Python distribution removal | #262 | **Stage 1 landed: the Rust core is opt-in behind the existing launcher.** Its #261 prerequisite — the production OwnIR executable alone (C-5) — is satisfied: 261.B landed (PR #347, `206e9c7`) and #261 is closed completed; #260 reached; #345 is not on this path. Stage 1 is **not** a cutover: Python remains the default and the reference on all four launcher surfaces (`owen`, `own-check.sh`, `own-check.ps1`, the Action), nothing public defaults to Rust, and Python distribution is untouched. What exists now: one explicit `--engine python|rust|compare` selector (D1), the ratified `OWEN_RUST_CORE` candidate locator with no discovery of any kind and a visible configuration failure (rc 2) when it cannot be used (D3/D3.1), an unexpected Rust child status mapped to the public internal-error path with the raw status retained in a typed `child_exit_code` at report schema 2 (D5), and a launcher-seam compare mode that extracts once, proves both engines received byte-identical input, and refuses to answer — public exit 5 with reproduction evidence — when they diverge or either fails (D4/D4.1). No silent fallback anywhere: a Rust failure is never a Python success. Engine selection stays outside `own-cli` (C-4). Compare is a development/CI seam, not yet a promised public feature. Counts are generated (`docs/generated/p022-stage1-mutations.md`). #263's baselines are the evidence prerequisite of the cutover decision, not a normative blocker. Launcher rulings recorded in #262: engine selection is the launcher's, never the executable's; an unexpected Rust child exit code outside the legal set takes the public internal-error path with the raw child status retained in the evidence; no silent fallback | -**Preferred queue:** **#262 is next.** #261's production OwnIR executable is +**Preferred queue:** **#262 is in progress — Stage 1 (opt-in Rust behind the launcher) landed; Stage 2 is next and needs its own authorization.** #261's production OwnIR executable is built and replaying on both platforms (row 7b), and #261 is closed completed (2026-09-08, PR #347), so the queue has moved past it. In parallel and off the critical chain: #257, #263 (the evidence prerequisite of #262's decision), #345 — the residual `.own`/dev diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 8eadea2a..bad0febf 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -41,7 +41,7 @@ proposal is marked `done` with a pointer. | [P-017](P-017-multi-stack-frontends.md) | Multi-stack frontends (OwnTS / OwnJVM: OwnJava + OwnKotlin) | draft | | [P-020](P-020-ownts-react-effects.md) | OwnTS React effects profile (`Own.React`) — the effect-storm angle | draft | | [P-021](P-021-async-audit-pack.md) | Async audit pack (`Own.Async`) | draft | -| [P-022](P-022-rust-core-migration.md) | Rust core migration: crate DAG, patterns, prior art, differential oracle (Python = golden) | in execution — steps 0–4 built (#214/#249); step 5a done (full diagnostic contract, #255 via #319/#320/#321); step 5b SARIF done (#256; `.ownreport.json` struck — a buffer report needing the AST, not a diagnostics surface); step 6a done (`spec/Bridge.md`, #258); step 6b complete at final acceptance (`own-lowered`/`own-bridge`, #259: lowering and MOS parity landed; strict-door validation complete with no known divergence — the first 0/0/0 proved to be the ledger agreeing with its own author, and the second omitted two families that a Python-first defensive-limit change (#326) had to close before the third could measure them; analysis wiring complete at the checkpoint-4 surface — `check_facts` through the real analyses, Layer 3 goldens built, with an executable exclusion ledger naming each declared boundary; **cp5 complete at its surface** — the replay compares EVERY `Finding` member (the BR-V4 wording matrix and the BR-V5 evidence slices included) and every refusal in full, and a second fixture family freezes the BR-V9 rendered surfaces byte for byte, all against goldens none of which was regenerated; **row 4b complete** — the obligation-protocol analysis (OBL001–005) is ported into `own-analysis`, its typed values come from the ONE grammar in `own-ir` that the strict door already delegated to, an analysis-level fact-parity family freezes every violation member with zero Python, the bridge maps BR-P3 in its BR-V1 place, and both protocol documents are promoted out of the exclusion ledger without regenerating either golden; **#259 final acceptance reached** — the last thing it owed was the coordinate-domain decision, and that landed Python-first: `spec/OwnIR.md` §4.2 bounds every `line` to `[0, 2147483647]` and every `column` to `[1, 2147483647]` (int32 is the line type of every consumer this project feeds; `0` stays legal as the reference's own absent sentinel), every line-bearing field is validated including the two §4.2 recorded as checked nowhere, the tolerant door degrades an out-of-domain coordinate rather than clamping it, the Rust door and bridge mirror all of it, and the four `verdict_boundary_*` controls are promoted out of the exclusion ledger — which now names only the two #294 OD-1 door controls, a declared boundary rather than open work. Not shadow mode, which is #260's acceptance. Every count is generated: `docs/generated/p022-cp1-census.md`, `docs/generated/p022-cp4-census.md`, `docs/generated/p022-coord-census.md`, `docs/generated/p022-cp5-inventory.md`, `docs/generated/p022-cp4b-mutations.md` and `docs/generated/p022-coord-mutations.md`); step 7a shadow-mode INFRASTRUCTURE complete (checkpoints 1–4: `ownlang/repro.py` + `own-shadow` — canonical same-input `OwnIR` identity, the reproduction-artifact format, the engine protocol, the `AnalysisTrace` (#269) with stable-ID normalization, first-divergence reduction), and #260's **acceptance decisions landed over the committed corpus**: the verdict layer is in reduction scope (the scope IS the layer order), acceptance is a field of its own beside the observation kind under a frozen `(layer, kind, class)` boundary policy the refusing engine declares structurally, canonical SARIF is compared as a DERIVED surface rather than a layer, artifact v3 attests the raw input and each engine's `consumed` (so the byte-level same-input invariant is proved rather than approximated by canonical identity), and a dev-only `own-shadow-engine` adapter plus a compare driver run the two engines over one byte sequence in CI. **#260's final acceptance is REACHED**: compare mode reports zero acceptance-unexplained over its full test matrix — the committed corpus, the C# samples, the `examples/` tree, the five pinned OSS repositories of #243 at their verified pins and the large-solution controls — at all three layers and on the derived SARIF, on byte-attested same input, with the two #294 OD-1 typed-door boundaries declared by policy. The sweep is ten documents over six targets, each extracted exactly once through `own-check.sh --emit-facts` and compared from those bytes; a repository is not covered because its extraction succeeded, so a run that compared zero documents fails, a declared target nothing reached fails, and the denominators are recorded per target. Taking the measurement found six harness defects and no engine divergence. Still **not** shadow mode achieved, **not** "P-022 done" and **not** "Rust is the default" — that is #262's cutover behind #261; a crash is never a fallback, Python stays the public engine, and no production behaviour changed. Every count is generated (`docs/generated/p022-shadow-sweep.md`, `docs/generated/p022-shadow-census.md`, `docs/generated/p022-shadow-mutations.md`), the decisions are recorded verbatim in [the owner-decision ledger](../notes/p022-shadow-infra-owner-decisions.md), and the records are [the sweep note](../notes/p022-shadow-sweep.md) and [the acceptance note](../notes/p022-shadow-acceptance.md); **step 7b 261.A ratified and 261.B built** — #261's production Rust OwnIR executable `own-cli ownir` exists behind the unchanged `owen` launcher and reproduces the reference's `ownir` contract (argument handling, display policy, stream separation, the four formats and every exit code) over a frozen CLI fixture replayed with zero Python on Linux and Windows CI; the top-level shell follows the `owen` convention as a parity surface of its own and everything after `ownir` is the reference's own behaviour, measured; a catchable panic is one actionable message and exit 70 and an uncatchable death a visible hard failure, both measured under an off-by-default `fault-injection` feature. Owner decisions C-1..C-5 were applied, not re-litigated. 261.B is built and replaying on both platforms, with the four rulings settled as follows: (2a) the `ownir_version` Version messages are **byte-parity** — that text is ours on both sides, so the divergence was a Rust bug and was fixed rather than declared, and a second repair pass re-took the measurement over value **classes** rather than four hand-picked values, fixing three more defects a single-key integer control could not reach plus a round-half-to-even tie found by a 200 000-double sweep (24/24 classes and 20 000 randomized documents byte-identical, with V1/V2/V4 declared and excluded); (2b) the JSON parser detail is a **declared typed boundary, CLI-B1** — the CLI-owned wrapper `{path}: error: {path} is not valid JSON: ` is pinned byte-exact and only the parser library's own text after it is declared, guarded by an executable `kind == Json` proof and a negative control that runs ONE case against two byte sequences so eligibility can only turn on the facts bytes, and a Json rejection that loses its internal prefix fails onto rc 70 rather than passing through; (1) invalid UTF-8 is a **declared defect of the Python reference**, excluded from the byte contract pending a Python-first hygiene tail (`UnicodeDecodeError` -> `OwnIRError` -> rc 2) to close before public cutover, recorded in #262 — the tracker of record — and mirrored in #250's Still missing list; (3) Windows is **A** canonical reference parity plus **B** Rust portability, with **C** native-Windows Python parity explicitly **NOT claimed** — the reference there emits cp1252/CRLF and can fail with `UnicodeEncodeError`, recorded in #262 — the tracker of record — as a behavior change rather than parity, and mirrored in #250's Still missing list. Every count is generated (`docs/generated/p022-cli-census.md`, `docs/generated/p022-cli-mutations.md`), the record is [the note](../notes/p022-cli-ownir.md). Nothing is wired, published or defaulted — that is #262, and #261 is closed completed (2026-09-08, PR #347); step 8 (#262) is next, its #261 prerequisite satisfied, with #263's baselines as the evidence prerequisite of its decision | +| [P-022](P-022-rust-core-migration.md) | Rust core migration: crate DAG, patterns, prior art, differential oracle (Python = golden) | in execution — steps 0–4 built (#214/#249); step 5a done (full diagnostic contract, #255 via #319/#320/#321); step 5b SARIF done (#256; `.ownreport.json` struck — a buffer report needing the AST, not a diagnostics surface); step 6a done (`spec/Bridge.md`, #258); step 6b complete at final acceptance (`own-lowered`/`own-bridge`, #259: lowering and MOS parity landed; strict-door validation complete with no known divergence — the first 0/0/0 proved to be the ledger agreeing with its own author, and the second omitted two families that a Python-first defensive-limit change (#326) had to close before the third could measure them; analysis wiring complete at the checkpoint-4 surface — `check_facts` through the real analyses, Layer 3 goldens built, with an executable exclusion ledger naming each declared boundary; **cp5 complete at its surface** — the replay compares EVERY `Finding` member (the BR-V4 wording matrix and the BR-V5 evidence slices included) and every refusal in full, and a second fixture family freezes the BR-V9 rendered surfaces byte for byte, all against goldens none of which was regenerated; **row 4b complete** — the obligation-protocol analysis (OBL001–005) is ported into `own-analysis`, its typed values come from the ONE grammar in `own-ir` that the strict door already delegated to, an analysis-level fact-parity family freezes every violation member with zero Python, the bridge maps BR-P3 in its BR-V1 place, and both protocol documents are promoted out of the exclusion ledger without regenerating either golden; **#259 final acceptance reached** — the last thing it owed was the coordinate-domain decision, and that landed Python-first: `spec/OwnIR.md` §4.2 bounds every `line` to `[0, 2147483647]` and every `column` to `[1, 2147483647]` (int32 is the line type of every consumer this project feeds; `0` stays legal as the reference's own absent sentinel), every line-bearing field is validated including the two §4.2 recorded as checked nowhere, the tolerant door degrades an out-of-domain coordinate rather than clamping it, the Rust door and bridge mirror all of it, and the four `verdict_boundary_*` controls are promoted out of the exclusion ledger — which now names only the two #294 OD-1 door controls, a declared boundary rather than open work. Not shadow mode, which is #260's acceptance. Every count is generated: `docs/generated/p022-cp1-census.md`, `docs/generated/p022-cp4-census.md`, `docs/generated/p022-coord-census.md`, `docs/generated/p022-cp5-inventory.md`, `docs/generated/p022-cp4b-mutations.md` and `docs/generated/p022-coord-mutations.md`); step 7a shadow-mode INFRASTRUCTURE complete (checkpoints 1–4: `ownlang/repro.py` + `own-shadow` — canonical same-input `OwnIR` identity, the reproduction-artifact format, the engine protocol, the `AnalysisTrace` (#269) with stable-ID normalization, first-divergence reduction), and #260's **acceptance decisions landed over the committed corpus**: the verdict layer is in reduction scope (the scope IS the layer order), acceptance is a field of its own beside the observation kind under a frozen `(layer, kind, class)` boundary policy the refusing engine declares structurally, canonical SARIF is compared as a DERIVED surface rather than a layer, artifact v3 attests the raw input and each engine's `consumed` (so the byte-level same-input invariant is proved rather than approximated by canonical identity), and a dev-only `own-shadow-engine` adapter plus a compare driver run the two engines over one byte sequence in CI. **#260's final acceptance is REACHED**: compare mode reports zero acceptance-unexplained over its full test matrix — the committed corpus, the C# samples, the `examples/` tree, the five pinned OSS repositories of #243 at their verified pins and the large-solution controls — at all three layers and on the derived SARIF, on byte-attested same input, with the two #294 OD-1 typed-door boundaries declared by policy. The sweep is ten documents over six targets, each extracted exactly once through `own-check.sh --emit-facts` and compared from those bytes; a repository is not covered because its extraction succeeded, so a run that compared zero documents fails, a declared target nothing reached fails, and the denominators are recorded per target. Taking the measurement found six harness defects and no engine divergence. Still **not** shadow mode achieved, **not** "P-022 done" and **not** "Rust is the default" — that is #262's cutover behind #261; a crash is never a fallback, Python stays the public engine, and no production behaviour changed. Every count is generated (`docs/generated/p022-shadow-sweep.md`, `docs/generated/p022-shadow-census.md`, `docs/generated/p022-shadow-mutations.md`), the decisions are recorded verbatim in [the owner-decision ledger](../notes/p022-shadow-infra-owner-decisions.md), and the records are [the sweep note](../notes/p022-shadow-sweep.md) and [the acceptance note](../notes/p022-shadow-acceptance.md); **step 7b 261.A ratified and 261.B built** — #261's production Rust OwnIR executable `own-cli ownir` exists behind the unchanged `owen` launcher and reproduces the reference's `ownir` contract (argument handling, display policy, stream separation, the four formats and every exit code) over a frozen CLI fixture replayed with zero Python on Linux and Windows CI; the top-level shell follows the `owen` convention as a parity surface of its own and everything after `ownir` is the reference's own behaviour, measured; a catchable panic is one actionable message and exit 70 and an uncatchable death a visible hard failure, both measured under an off-by-default `fault-injection` feature. Owner decisions C-1..C-5 were applied, not re-litigated. 261.B is built and replaying on both platforms, with the four rulings settled as follows: (2a) the `ownir_version` Version messages are **byte-parity** — that text is ours on both sides, so the divergence was a Rust bug and was fixed rather than declared, and a second repair pass re-took the measurement over value **classes** rather than four hand-picked values, fixing three more defects a single-key integer control could not reach plus a round-half-to-even tie found by a 200 000-double sweep (24/24 classes and 20 000 randomized documents byte-identical, with V1/V2/V4 declared and excluded); (2b) the JSON parser detail is a **declared typed boundary, CLI-B1** — the CLI-owned wrapper `{path}: error: {path} is not valid JSON: ` is pinned byte-exact and only the parser library's own text after it is declared, guarded by an executable `kind == Json` proof and a negative control that runs ONE case against two byte sequences so eligibility can only turn on the facts bytes, and a Json rejection that loses its internal prefix fails onto rc 70 rather than passing through; (1) invalid UTF-8 is a **declared defect of the Python reference**, excluded from the byte contract pending a Python-first hygiene tail (`UnicodeDecodeError` -> `OwnIRError` -> rc 2) to close before public cutover, recorded in #262 — the tracker of record — and mirrored in #250's Still missing list; (3) Windows is **A** canonical reference parity plus **B** Rust portability, with **C** native-Windows Python parity explicitly **NOT claimed** — the reference there emits cp1252/CRLF and can fail with `UnicodeEncodeError`, recorded in #262 — the tracker of record — as a behavior change rather than parity, and mirrored in #250's Still missing list. Every count is generated (`docs/generated/p022-cli-census.md`, `docs/generated/p022-cli-mutations.md`), the record is [the note](../notes/p022-cli-ownir.md). Nothing was wired, published or defaulted by #261 — that is #262, and #261 is closed completed (2026-09-08, PR #347); **step 8 (#262) is in progress: Stage 1 landed, making the Rust core opt-in behind the existing launcher** via one explicit `--engine python|rust|compare` selector on all four launcher surfaces, the ratified `OWEN_RUST_CORE` candidate locator (no discovery; an unusable locator is a configuration failure, never a fallback), an unexpected Rust child status mapped to the public internal-error path with the raw status retained in a typed `child_exit_code`, and a launcher-seam compare mode that extracts once and refuses to answer when the engines disagree. Python remains the default and the reference, nothing public defaults to Rust, and Python distribution is untouched — Stage 1 is explicitly not the cutover, which stays behind Gate G3. Its #261 prerequisite is satisfied, with #263's baselines as the evidence prerequisite of its decision | | [P-023](P-023-architecture-guard.md) | Architecture guard (`Own.Arch`): rules.yaml intent model + dependency-graph gate + baseline ratchet | draft | | [P-024](P-024-security-audit-profile.md) | Security audit profile (external tools + SARIF adapters; rejects own scanner engine) | draft | | [P-025](P-025-obligation-protocols.md) | Obligation protocols (`Own.Protocols`): barrier-sensitive project invariants (OBL001–005) | first slice built (core + bridge + fixtures; extractor pending) | diff --git a/frontend/roslyn/OwnSharp.Cli/README.md b/frontend/roslyn/OwnSharp.Cli/README.md index 93cbef85..ad29ce3b 100644 --- a/frontend/roslyn/OwnSharp.Cli/README.md +++ b/frontend/roslyn/OwnSharp.Cli/README.md @@ -63,12 +63,48 @@ owen check . --format sarif > owen.sarif # feed github/codeql-ac Uninstall/upgrade: `dotnet tool uninstall --global Owen.Cli`, then reinstall as above (bump `--version` if you rebuilt with a new ``). +## Engine selection (#262 Stage 1) + +Python is the **default** and the reference. `--engine` selects otherwise: + +| Value | What runs | +|---|---| +| `python` | the vendored Python core — the default and the reference | +| `rust` | the Rust core (`own-cli ownir`) instead | +| `compare` | both, over one captured input, reporting the reference's result only when they agree byte for byte | + +`rust` and `compare` need the candidate binary's absolute path in +**`OWEN_RUST_CORE`**. There is deliberately **no discovery** — no `PATH` +lookup, no `rust/target/` probing, no "first binary found" — because discovery +is how a stale binary silently stands in for the one you meant to test. A +missing, empty, nonexistent, non-file or non-executable `OWEN_RUST_CORE` is a +**configuration error (exit 2)** with one actionable message, and it never +falls back to Python. + +There is no silent fallback anywhere: a Rust failure is never turned into a +Python success. An unexpected Rust child status becomes owen's internal-error +exit (`5`) with the raw status kept in the diagnostic report's +`child_exit_code`, rather than escaping as a meaningless number. + +`compare` is a development/CI seam for the migration, **not yet a promised +public feature**. When the engines disagree — or when either fails — owen +exits `5` with reproduction evidence rather than picking a winner: a reference +and a candidate that disagree mean owen cannot honestly emit one answer. On +native Windows the Python reference emits cp1252/CRLF where the Rust core +emits canonical UTF-8, so `compare` will report a real divergence there for +non-ASCII output; that is #262's declared Windows behaviour change being +visible, not a defect. + +Rollback is explicit: select `--engine python` (or simply stop passing +`--engine`). Nothing about Stage 1 moves the public default. + ## Flags (mirror `scripts/own-check.sh` 1:1) | Flag | Default | | |---|---|---| | `--format {human,github,msbuild,sarif}` | `human` | finding surface | | `--severity {error,warning}` | `error` | how findings are shown | +| `--engine {python,rust,compare}` | `python` | which analysis engine runs (#262 Stage 1) | | `--fail-on-finding` | off | exit with the core's code (1 = findings) instead of always 0 | | `--emit-facts ` | — | also write the intermediate OwnIR facts.json | | `--legacy` | off | flat name-based local-`IDisposable` detector instead of `--flow-locals` | @@ -76,7 +112,8 @@ as above (bump `--version` if you rebuilt with a new ``). | `--body-throw-edges` | off | opt-in: flag body-level (no-`try`) dispose-not-called-on-throw | Exit codes: `0` clean, `1` findings (only with `--fail-on-finding`), `2` a -usage or contract error (bad flags, bad facts, a drifted contract), `3` no +usage or contract error (bad flags, bad facts, a drifted contract, or an +unusable `OWEN_RUST_CORE` under `--engine rust|compare`), `3` no usable Python found, `4` no supported input found (nothing matching the included frontend), `5` an **internal error** — a bug in owen or a stage it drives (extractor/core crash). An internal error is never silence, never a diff --git a/rust/README.md b/rust/README.md index 6bb95dbf..2bf93161 100644 --- a/rust/README.md +++ b/rust/README.md @@ -29,7 +29,7 @@ implementation-status block; this table is the one-line orientation: | `own-lowered` | **done** (#259) | The typed Layer 2 document + canonical emitter the bridge lowers into. | | `own-bridge` | **done** (#259 final acceptance reached, PR #341; the exclusion ledger names only the two #294 OD-1 door controls, a declared boundary) | The OwnIR bridge: facts → Layer 2 → core AST → analyses → verdicts (`lower`, `dump_summaries`, `check_facts`). | | `own-codegen` | not started (#257) | C# emission (`emit_*` templates), verdict-independent. | -| `own-cli` | built for the `ownir` slice (#261 261.B landed; #261 closed completed) | The entry-point binary: `own-cli ownir [--format F] [--severity S] [--verbosity V]`, the one core invocation the product seam makes today, reproducing the reference over a frozen fixture replayed with **zero Python** on Linux and Windows CI. One engine, no engine selection, no fallback: it knows nothing of Python, and it is not wired to anything — `owen`, `own-check.*` and the Action are untouched, and Python stays the public engine until #262. The top-level shell follows the public `owen` convention as a parity surface of its own; everything after `ownir` is the reference as measured. The strict door's Version messages are byte-parity with the reference, measured over value **classes** rather than sample values — 24/24 classes and 20 000 randomized documents identical, with three exceptions named and excluded rather than counted (the reference's non-standard JSON constants, the literal `-0`, and the two sides' independently versioned Unicode tables); the JSON parser's own detail is the one declared boundary (CLI-B1), whose CLI-owned wrapper is pinned and whose guard proves `kind == Json` before relaxing anything, on facts bytes supplied to it so its negative control flips on content alone. An off-by-default `fault-injection` feature carries the two forced-failure controls (`cargo test -p own-cli --features fault-injection --test faults`) so the panic and death rulings are measured, not asserted. The residual `.own`/dev subcommands (`cfg`, `summaries`, `explain`, `check`; `emit` after #257) are #345 and join THIS binary. `own-oracle` is the dev-only differential harness alongside it. | +| `own-cli` | built for the `ownir` slice (#261 261.B landed; #261 closed completed) | The entry-point binary: `own-cli ownir [--format F] [--severity S] [--verbosity V]`, the one core invocation the product seam makes today, reproducing the reference over a frozen fixture replayed with **zero Python** on Linux and Windows CI. One engine, no engine selection, no fallback: it knows nothing of Python. #262's Stage 1 wires it in as an **opt-in** engine — the LAUNCHER selects it with `--engine rust|compare` and locates this binary through `OWEN_RUST_CORE` — and that selector deliberately lives outside this crate (C-4): nothing here gained an engine flag or any notion of Python. Python remains the default and the reference on every launcher surface; nothing public defaults to Rust until #262's Gate G3. The top-level shell follows the public `owen` convention as a parity surface of its own; everything after `ownir` is the reference as measured. The strict door's Version messages are byte-parity with the reference, measured over value **classes** rather than sample values — 24/24 classes and 20 000 randomized documents identical, with three exceptions named and excluded rather than counted (the reference's non-standard JSON constants, the literal `-0`, and the two sides' independently versioned Unicode tables); the JSON parser's own detail is the one declared boundary (CLI-B1), whose CLI-owned wrapper is pinned and whose guard proves `kind == Json` before relaxing anything, on facts bytes supplied to it so its negative control flips on content alone. An off-by-default `fault-injection` feature carries the two forced-failure controls (`cargo test -p own-cli --features fault-injection --test faults`) so the panic and death rulings are measured, not asserted. The residual `.own`/dev subcommands (`cfg`, `summaries`, `explain`, `check`; `emit` after #257) are #345 and join THIS binary. `own-oracle` is the dev-only differential harness alongside it. | ## Build & test diff --git a/scripts/render_checkpoint_status.py b/scripts/render_checkpoint_status.py index dcc75f43..450f3adb 100644 --- a/scripts/render_checkpoint_status.py +++ b/scripts/render_checkpoint_status.py @@ -134,6 +134,7 @@ MUTATIONS_MD = "p022-cp4-mutations.md" CLI_CENSUS_MD = "p022-cli-census.md" CLI_MUTATIONS_MD = "p022-cli-mutations.md" +STAGE1_MUTATIONS_MD = "p022-stage1-mutations.md" SHADOW_CENSUS_MD = "p022-shadow-census.md" SHADOW_MUTATIONS_MD = "p022-shadow-mutations.md" SHADOW_SWEEP_MD = "p022-shadow-sweep.md" @@ -184,6 +185,15 @@ ("261.B — `own-cli ownir`: the display policy, the CLI's SARIF bytes, the " "usage exit codes and the process contract", "p022-cli-1"), ) +# P-022 step 8 (#262) STAGE 1: the launcher's engine-selection contract. One +# campaign, because the surface is one seam — the selector, the candidate +# locator, the child-status mapping and the compare contract are read together +# and fail together. +STAGE1_CAMPAIGNS = ( + ("Stage 1 — the launcher's `--engine` contract: the default, the candidate " + "locator, the Rust child status and the compare result contract", + "p022-stage1-1"), +) SELF = "scripts/render_checkpoint_status.py" @@ -1188,6 +1198,27 @@ def fragments() -> tuple[dict[str, str], list[str]]: CLI_CAMPAIGNS) out[CLI_MUTATIONS_MD] = cli problems.extend(f"mutation campaign {p}" for p in cli_problems) + stage1, stage1_problems = render_campaign_set( + "# P-022 step 8 (#262) Stage 1 — mutation campaigns", + "Stage 1 makes the Rust core SELECTABLE by the launcher while Python stays the " + "default and the reference. Every mutation below is a plausible MISREADING of " + "that contract rather than a syntactic accident: the default moved because the " + "cutover was read as already decided; Python resolved for every engine because " + "the old unconditional resolution looked harmless; 70 admitted to the verdict " + "set because both engines document it; an unexpected child status propagated " + "as itself because that looked like faithfulness; the shell falling back to " + "Python because answering the user looked helpful. Each would pass a reviewer " + "who had read the stage's summary instead of its rulings. Every mutation edits " + "a **production** launcher surface (P-022 discipline 2), and the single layer " + "REBUILDS the launcher before testing it — a mutated `.cs` file is otherwise " + "invisible to controls that drive a compiled binary — and runs every control " + "for every mutation with no fail-fast (discipline 3), under " + "`OWEN_STAGE1_REQUIRE=1` so a control that could not run is a failure rather " + "than a silently shrinking denominator. The counts are derived from the " + "recorded run by `scripts/mutate_campaign.summarize()`, never typed.", + STAGE1_CAMPAIGNS) + out[STAGE1_MUTATIONS_MD] = stage1 + problems.extend(f"mutation campaign {p}" for p in stage1_problems) return out, problems @@ -1233,8 +1264,8 @@ def main(argv: list[str]) -> int: if argv: print(f"checkpoint status fragments OK: {CENSUS_MD}, {CP1_CENSUS_MD}, " f"{COORD_CENSUS_MD}, {INVENTORY_MD}, {MUTATIONS_MD}, {CP5_MUTATIONS_MD}, " - f"{SHADOW_CENSUS_MD}, {SHADOW_MUTATIONS_MD}, {SHADOW_SWEEP_MD} in " - f"sync with the evidence") + f"{SHADOW_CENSUS_MD}, {SHADOW_MUTATIONS_MD}, {SHADOW_SWEEP_MD}, " + f"{STAGE1_MUTATIONS_MD} in sync with the evidence") return 0 From 757c29d88edbb8eff5f9a4d870725c692b98e867 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 01:40:12 +0000 Subject: [PATCH 03/22] test(stage1): make the engine controls load-bearing on both launcher surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first mutation campaign over these surfaces caught 11 of 16 and named the gap precisely: five mutations survived, and one was caught by a control that was not supposed to be its catcher. The cause was one hole, not five — the compare and locator controls drove own-check.sh but never the `owen` launcher, so every C#-side misreading was invisible to them: M06/M07 the launcher's own locator exit code moved to 5 / to 3, unobserved M11 the launcher's compare skipped the execution-failure check M12 the launcher's compare judged a zero-document input anyway M10 the launcher's compare exposed the reference's result on divergence (caught only incidentally, by the same-input control) Every affected control now drives BOTH surfaces, which is also what D2 actually asks for: the engine-selection semantics are one contract, so a control that only ever exercises one surface is not measuring the contract. M14 survived for a different reason worth recording: the diverging stub differed from the reference in its EXIT CODE as well as its bytes, so a compare that had stopped comparing stdout still looked correct — the exit check alone flagged it. The stub now returns the same exit code the reference does and differs only in the bytes, so the stdout comparison is load-bearing. Strengthening the controls then found a real production inconsistency rather than a test bug: own-check.sh names the input digest and the candidate on stderr, but the launcher printed only the evidence file's path. A reader who must open a JSON file to learn the two identities that make a compare reproducible has been handed a filename, not a reproduction — so the launcher now prints the same two facts, and the surfaces say one thing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 144 ++++++++++++++ frontend/roslyn/OwnSharp.Cli/CompareMode.cs | 9 + tests/test_stage1_engine.py | 205 +++++++++++++------- 3 files changed, 292 insertions(+), 66 deletions(-) create mode 100644 docs/evidence/p022-stage1-1.result.json diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json new file mode 100644 index 00000000..5c76f45c --- /dev/null +++ b/docs/evidence/p022-stage1-1.result.json @@ -0,0 +1,144 @@ +{ + "schema": 1, + "comment": "Recorded mutation-campaign run (scripts/mutate_campaign.py --run). Raw facts only: outcomes, catchers, provenance. Counts are derived by scripts/render_checkpoint_status.py; regenerate this file by re-running the campaign, never by hand.", + "campaign": "p022-stage1-1", + "definition": "docs/evidence/p022-stage1-1.json", + "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", + "source_commit": "9f34e2058de42ec8464a72b0180bda011a541a08", + "dirty": false, + "recorded_at": "2026-09-09T01:37:09Z", + "layers": [ + "stage1" + ], + "command": "every layer the definition declares, for every mutation", + "control": { + "id": "M00", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 30.4 + }, + "mutations": [ + { + "id": "M01", + "outcome": "caught", + "catchers": [ + "stage1::default-stays-python" + ], + "elapsed_seconds": 24.9 + }, + { + "id": "M02", + "outcome": "caught", + "catchers": [ + "stage1::rust-actually-runs-rust" + ], + "elapsed_seconds": 23.2 + }, + { + "id": "M03", + "outcome": "caught", + "catchers": [ + "stage1::rc70-is-not-a-verdict", + "stage1::rust-failure-no-fallback" + ], + "elapsed_seconds": 24.3 + }, + { + "id": "M04", + "outcome": "caught", + "catchers": [ + "stage1::raw-rc-retained", + "stage1::rc70-is-not-a-verdict", + "stage1::rust-failure-no-fallback", + "stage1::unexpected-rc-maps-to-5" + ], + "elapsed_seconds": 23.2 + }, + { + "id": "M05", + "outcome": "caught", + "catchers": [ + "stage1::raw-rc-retained" + ], + "elapsed_seconds": 23.9 + }, + { + "id": "M06", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 24.0 + }, + { + "id": "M07", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 23.6 + }, + { + "id": "M08", + "outcome": "caught", + "catchers": [ + "stage1::rust-failure-no-fallback" + ], + "elapsed_seconds": 25.1 + }, + { + "id": "M09", + "outcome": "caught", + "catchers": [ + "stage1::bad-locator-is-2" + ], + "elapsed_seconds": 25.2 + }, + { + "id": "M10", + "outcome": "caught", + "catchers": [ + "stage1::compare-same-input" + ], + "elapsed_seconds": 23.9 + }, + { + "id": "M11", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 24.9 + }, + { + "id": "M12", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 23.7 + }, + { + "id": "M13", + "outcome": "caught", + "catchers": [ + "stage1::candidate-identity" + ], + "elapsed_seconds": 24.6 + }, + { + "id": "M14", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 24.4 + }, + { + "id": "M15", + "outcome": "caught", + "catchers": [ + "stage1::divergence-is-5" + ], + "elapsed_seconds": 24.0 + }, + { + "id": "M16", + "outcome": "caught", + "catchers": [ + "stage1::compare-zero-document" + ], + "elapsed_seconds": 24.1 + } + ] +} diff --git a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs index 1a2f5e32..a9d78756 100644 --- a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs +++ b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs @@ -187,6 +187,15 @@ int Fail(string[] a, RustCore core, EngineOutcome? py, EngineOutcome? rs, string diagnostic, int? childExitCode) { Console.Error.WriteLine($"owen: --engine compare: {diagnostic}"); + // The reproduction line is on STDERR, not only inside the evidence + // file: the two identities that make a compare reproducible are the + // input bytes and the candidate binary, and a reader who has to + // open a JSON file to learn them has been handed a filename rather + // than a reproduction. own-check.sh prints the same two facts, so + // the surfaces say one thing (D2). + Console.Error.WriteLine( + $" Reproduction — input sha256 {(capturedSha.Length > 0 ? capturedSha : "(no capture)")}, " + + $"candidate {core.Path} (sha256 {core.Sha256})"); var path = WriteEvidence(a, core, capturedSha, py, rs, verdict: childExitCode is null && py is not null && rs is not null ? "divergence" diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index 3648ad67..053e070d 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -207,30 +207,49 @@ def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: "non-executable": str(not_exec), } problems = [] - for engine in ("rust", "compare"): - for name, value in cases.items(): - env = {"OWEN_RUST_CORE": value} if value is not None else {} - e = dict(os.environ) - e.update(env) - if value is None: - e.pop("OWEN_RUST_CORE", None) - r = subprocess.run( - ["bash", str(ROOT / "scripts/own-check.sh"), - "--engine", engine, "--", str(sample)], - capture_output=True, env=e, cwd=str(ROOT), check=False) - if r.returncode != 2: - problems.append(f"{engine}/{name}: exit {r.returncode}, expected 2") - merged = (r.stdout + r.stderr).decode("utf-8", "replace") - # A fallback would have produced a verdict; the diagnostic must - # also say, in as many words, that no fallback happened. - if "finding" in merged and "OWEN_RUST_CORE" not in merged: - problems.append(f"{engine}/{name}: produced a verdict — looks like a fallback") - if r.returncode == 2 and "did not fall back to Python" not in merged: - problems.append(f"{engine}/{name}: diagnostic does not deny a Python fallback") + # D2: the locator contract is ONE contract, so it is driven on BOTH the + # shell surface and the `owen` launcher. Testing only the shell would let + # the launcher's own exit code drift freely — which is exactly what a + # campaign mutation of that constant proved. + surfaces: list[tuple[str, object]] = [("own-check.sh", "shell")] + if launcher_dll() is not None and have_dotnet(): + surfaces.append(("owen", "launcher")) + else: + skip(check + "/owen", "no built launcher/dotnet") + + for surface_name, kind in surfaces: + for engine in ("rust", "compare"): + for name, value in cases.items(): + env = {"OWEN_RUST_CORE": value} if value is not None else {} + e = dict(os.environ) + e.update(env) + if value is None: + e.pop("OWEN_RUST_CORE", None) + if kind == "shell": + r = subprocess.run( + ["bash", str(ROOT / "scripts/own-check.sh"), + "--engine", engine, "--", str(sample)], + capture_output=True, env=e, cwd=str(ROOT), check=False) + else: + r = subprocess.run( + ["dotnet", str(launcher_dll()), "check", "--engine", engine, str(sample)], + capture_output=True, env=e, cwd=str(ROOT), check=False) + where = f"{surface_name}/{engine}/{name}" + if r.returncode != 2: + problems.append(f"{where}: exit {r.returncode}, expected 2 " + "(not 3 — that is Python-specific; not 5 — that is an " + "internal failure)") + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + # A fallback would have produced a verdict; the diagnostic must + # also say, in as many words, that no fallback happened. + if "finding" in merged and "OWEN_RUST_CORE" not in merged: + problems.append(f"{where}: produced a verdict — looks like a fallback") + if r.returncode == 2 and "did not fall back to Python" not in merged: + problems.append(f"{where}: diagnostic does not deny a Python fallback") if problems: fail(check, "; ".join(problems)) else: - ok(check, f"{len(cases) * 2} invalid-locator cases all exit 2, no fallback") + ok(check, f"{len(cases) * 2 * len(surfaces)} invalid-locator cases all exit 2, no fallback") def control_default_stays_python(sample: Path) -> None: @@ -552,24 +571,38 @@ def control_compare_zero_document(tmp: Path) -> None: empty_dir = tmp / "empty-sample" empty_dir.mkdir(exist_ok=True) (empty_dir / "Nothing.cs").write_text(EMPTY_CS, encoding="utf-8") - r = run_own_check(["--engine", "compare", "--format", "human", "--", str(empty_dir)], - env={"OWEN_RUST_CORE": core}) - merged = (r.stdout + r.stderr).decode("utf-8", "replace") - if r.returncode in (0, 1): - fail(check, f"a zero-document compare exited {r.returncode} — it passed instead of failing") - return - if r.returncode != 5: - # Exit 4 (no supported input) is a different, legitimate refusal: the - # sample never reached the engines at all, so the control cannot speak. + runs = [("own-check.sh", + run_own_check(["--engine", "compare", "--format", "human", "--", str(empty_dir)], + env={"OWEN_RUST_CORE": core}))] + # D2 again: the guard belongs to both surfaces, and a C#-side mutation is + # invisible to a shell-only control. + owen = run_owen(["--engine", "compare", "--format", "human", str(empty_dir)], + env={"OWEN_RUST_CORE": core}) + if owen is not None: + runs.append(("owen", owen)) + else: + skip(check + "/owen", "no built launcher/dotnet") + + problems = [] + for where, r in runs: + merged = (r.stdout + r.stderr).decode("utf-8", "replace") if r.returncode == 4: - skip(check, "the sample was rejected as unsupported input before the engines ran") + # Exit 4 (no supported input) is a different, legitimate refusal: + # the sample never reached the engines, so the control cannot speak. + skip(check, f"{where}: the sample was rejected as unsupported input before the " + "engines ran") return - fail(check, f"a zero-document compare exited {r.returncode}, expected 5") - return - if "nothing to analyse" not in merged and "zero document" not in merged: - fail(check, "a zero-document compare failed without saying why") + if r.returncode in (0, 1): + problems.append(f"{where}: a zero-document compare exited {r.returncode} — it " + "passed instead of failing") + elif r.returncode != 5: + problems.append(f"{where}: a zero-document compare exited {r.returncode}, expected 5") + elif "nothing to analyse" not in merged: + problems.append(f"{where}: a zero-document compare failed without saying why") + if problems: + fail(check, "; ".join(problems)) return - ok(check, "a zero-document compare fails (exit 5) and says so") + ok(check, "a zero-document compare fails (exit 5) and says so, on every surface") def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: @@ -588,43 +621,83 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: skip(c, "no dotnet") return - # (b) divergence: a candidate that answers legally but differently. - diverging = write_stub(tmp / "diverging-core", 0, stdout="a different answer\n") - r = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], - env={"OWEN_RUST_CORE": str(diverging)}) - merged = (r.stdout + r.stderr).decode("utf-8", "replace") - if r.returncode == 1: - fail(div_check, "a compare divergence exited 1 — in public Owen that already means findings") - elif r.returncode != 5: - fail(div_check, f"a compare divergence exited {r.returncode}, expected public 5") - elif "divergence" not in merged: - fail(div_check, "a compare divergence exited 5 without an actionable diagnostic") - elif "sha256" not in merged: - fail(div_check, "a compare divergence produced no reproduction evidence (no input digest)") + # (b) divergence. The candidate answers LEGALLY and with the SAME exit code + # the reference produces (1 = findings on this leaky sample), differing only + # in the bytes. That is deliberate: a stub that also differed in its exit + # code would let a compare that had stopped comparing stdout still look + # correct, because the exit-code check alone would flag the divergence. + diverging = write_stub(tmp / "diverging-core", 1, stdout="a different answer\n") + div_runs = [("own-check.sh", + run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], + env={"OWEN_RUST_CORE": str(diverging)}))] + owen_div = run_owen(["--engine", "compare", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": str(diverging)}) + if owen_div is not None: + div_runs.append(("owen", owen_div)) else: - ok(div_check, "a compare divergence is public exit 5 with reproduction evidence") - - if b"a different answer" in r.stdout: - fail(sub_check, "the candidate's answer was exposed as the result of a diverging compare") + skip(div_check + "/owen", "no built launcher/dotnet") + + div_problems, sub_problems = [], [] + for where, r in div_runs: + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode == 1: + div_problems.append(f"{where}: a compare divergence exited 1 — in public Owen that " + "already means findings") + elif r.returncode != 5: + div_problems.append(f"{where}: a compare divergence exited {r.returncode}, " + "expected public 5") + elif "divergence" not in merged: + div_problems.append(f"{where}: a compare divergence exited 5 without an actionable " + "diagnostic") + elif "sha256" not in merged: + div_problems.append(f"{where}: a compare divergence produced no reproduction " + "evidence (no input digest)") + if b"a different answer" in r.stdout: + sub_problems.append(f"{where}: the candidate's answer was exposed as the result of " + "a diverging compare") + if div_problems: + fail(div_check, "; ".join(div_problems)) else: - ok(sub_check, "no engine's answer was exposed on divergence") + ok(div_check, "a divergence in the bytes alone is public exit 5 with reproduction " + "evidence, on every surface") # (c) execution failure: a candidate that produces no verdict at all. crashing = write_stub(tmp / "crashing-core", 42, stderr="forced execution failure\n") - r2 = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], - env={"OWEN_RUST_CORE": str(crashing)}) - merged2 = (r2.stdout + r2.stderr).decode("utf-8", "replace") - if r2.returncode != 5: - fail(exec_check, f"a compare execution failure exited {r2.returncode}, expected public 5") - elif "execution failure" not in merged2: - fail(exec_check, "a compare execution failure exited 5 without an actionable diagnostic") - elif "42" not in merged2: - fail(exec_check, "a compare execution failure did not retain the raw Rust child status") + exec_runs = [("own-check.sh", + run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], + env={"OWEN_RUST_CORE": str(crashing)}))] + owen_exec = run_owen(["--engine", "compare", "--format", "human", str(sample)], + env={"OWEN_RUST_CORE": str(crashing)}) + if owen_exec is not None: + exec_runs.append(("owen", owen_exec)) else: - ok(exec_check, "a compare execution failure is public exit 5 with failure evidence") + skip(exec_check + "/owen", "no built launcher/dotnet") + + exec_problems = [] + for where, r in exec_runs: + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode != 5: + exec_problems.append(f"{where}: a compare execution failure exited {r.returncode}, " + "expected public 5") + elif "execution failure" not in merged: + exec_problems.append(f"{where}: a compare execution failure exited 5 without an " + "actionable diagnostic") + elif "42" not in merged: + exec_problems.append(f"{where}: a compare execution failure did not retain the raw " + "Rust child status") + if b"OWN001" in r.stdout: + sub_problems.append(f"{where}: Python's verdict was exposed after the candidate " + "failed the compare") + if exec_problems: + fail(exec_check, "; ".join(exec_problems)) + else: + ok(exec_check, "a compare execution failure is public exit 5 with failure evidence, on " + "every surface") - if b"OWN001" in r2.stdout: - fail(sub_check, "Python's verdict was exposed after the candidate failed the compare") + if sub_problems: + fail(sub_check, "; ".join(sub_problems)) + else: + ok(sub_check, "no engine's answer was exposed on divergence or on failure") def control_candidate_identity(sample: Path, tmp: Path) -> None: From 1e89d8aea28d75bc7b1f15576ee7893cfe1eadcb Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 01:49:49 +0000 Subject: [PATCH 04/22] test(stage1): D4.1(b) exposes NEITHER engine's verdict on divergence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The second campaign caught 16/16 but still reported M10's expected catchers as missed: the mutation that exposes the REFERENCE's result on a divergence tripped `divergence-is-5` but not `compare-no-substitution`. The tempting fix was to drop that expectation, since "substitution" reads naturally as one engine's answer standing in for the other's FAILURE. Reading the ruling instead of the word settles it the other way: D4.1(b) says no engine's verdict is exposed as the authoritative result. Owen cannot honestly emit one answer while its reference and its candidate contradict each other, so falling back on "Python is the reference, trust it" is the same failure as trusting the candidate — it merely feels safer. So the control is strengthened rather than the expectation lowered: on divergence it now also fails if the reference's verdict reaches stdout. The campaign expectation stands as authored. Also records the second campaign's result, superseded by the run this commit's tree produces. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 71 ++++++++++++++----------- tests/test_stage1_engine.py | 9 ++++ 2 files changed, 50 insertions(+), 30 deletions(-) diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index 5c76f45c..503cded7 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -4,9 +4,9 @@ "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", - "source_commit": "9f34e2058de42ec8464a72b0180bda011a541a08", + "source_commit": "757c29d88edbb8eff5f9a4d870725c692b98e867", "dirty": false, - "recorded_at": "2026-09-09T01:37:09Z", + "recorded_at": "2026-09-09T01:48:28Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 30.4 + "elapsed_seconds": 34.5 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 24.9 + "elapsed_seconds": 29.4 }, { "id": "M02", @@ -32,7 +32,7 @@ "catchers": [ "stage1::rust-actually-runs-rust" ], - "elapsed_seconds": 23.2 + "elapsed_seconds": 26.8 }, { "id": "M03", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 24.3 + "elapsed_seconds": 28.0 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 23.2 + "elapsed_seconds": 27.9 }, { "id": "M05", @@ -60,19 +60,23 @@ "catchers": [ "stage1::raw-rc-retained" ], - "elapsed_seconds": 23.9 + "elapsed_seconds": 28.9 }, { "id": "M06", - "outcome": "survived", - "catchers": [], - "elapsed_seconds": 24.0 + "outcome": "caught", + "catchers": [ + "stage1::bad-locator-is-2" + ], + "elapsed_seconds": 27.9 }, { "id": "M07", - "outcome": "survived", - "catchers": [], - "elapsed_seconds": 23.6 + "outcome": "caught", + "catchers": [ + "stage1::bad-locator-is-2" + ], + "elapsed_seconds": 28.3 }, { "id": "M08", @@ -80,7 +84,7 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 25.1 + "elapsed_seconds": 28.2 }, { "id": "M09", @@ -88,27 +92,32 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 25.2 + "elapsed_seconds": 28.6 }, { "id": "M10", "outcome": "caught", "catchers": [ - "stage1::compare-same-input" + "stage1::compare-same-input", + "stage1::divergence-is-5" ], - "elapsed_seconds": 23.9 + "elapsed_seconds": 28.5 }, { "id": "M11", - "outcome": "survived", - "catchers": [], - "elapsed_seconds": 24.9 + "outcome": "caught", + "catchers": [ + "stage1::exec-failure-is-5" + ], + "elapsed_seconds": 28.1 }, { "id": "M12", - "outcome": "survived", - "catchers": [], - "elapsed_seconds": 23.7 + "outcome": "caught", + "catchers": [ + "stage1::compare-zero-document" + ], + "elapsed_seconds": 28.6 }, { "id": "M13", @@ -116,13 +125,15 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 24.6 + "elapsed_seconds": 28.3 }, { "id": "M14", - "outcome": "survived", - "catchers": [], - "elapsed_seconds": 24.4 + "outcome": "caught", + "catchers": [ + "stage1::divergence-is-5" + ], + "elapsed_seconds": 28.0 }, { "id": "M15", @@ -130,7 +141,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 24.0 + "elapsed_seconds": 28.6 }, { "id": "M16", @@ -138,7 +149,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 24.1 + "elapsed_seconds": 29.0 } ] } diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index 053e070d..b5a51cd6 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -652,9 +652,18 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: elif "sha256" not in merged: div_problems.append(f"{where}: a compare divergence produced no reproduction " "evidence (no input digest)") + # D4.1 (b) is stricter than "do not prefer the candidate": when the + # engines disagree, NO engine's verdict is exposed as the authoritative + # result. Owen cannot honestly emit one answer while its reference and + # its candidate contradict each other, so falling back on "Python is + # the reference, trust it" is the same failure as trusting the + # candidate — it just feels safer. if b"a different answer" in r.stdout: sub_problems.append(f"{where}: the candidate's answer was exposed as the result of " "a diverging compare") + if b"OWN001" in r.stdout: + sub_problems.append(f"{where}: the reference's verdict was exposed as the result of " + "a diverging compare — D4.1(b) exposes NEITHER engine's") if div_problems: fail(div_check, "; ".join(div_problems)) else: From 063ac628d8d417fa12c4b02f364b29c4324395bb Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 01:59:42 +0000 Subject: [PATCH 05/22] style(stage1): wrap two over-long lines to satisfy the repo's ruff gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pure formatting — no control changed meaning. Caught by running the repo's own lint gate locally rather than by discovering it in CI. The campaign is re-run on top of this commit so the recorded provenance names the tree that actually ships: the previous run measured a control file two line-wraps different from this one, and evidence that names a tree nobody can check out is weaker than evidence that names one they can. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 38 +++++++++++----------- docs/generated/p022-stage1-mutations.md | 43 +++++++++++++++++++++++++ tests/test_stage1_engine.py | 6 ++-- 3 files changed, 67 insertions(+), 20 deletions(-) create mode 100644 docs/generated/p022-stage1-mutations.md diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index 503cded7..2514dc9b 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -4,9 +4,9 @@ "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", - "source_commit": "757c29d88edbb8eff5f9a4d870725c692b98e867", + "source_commit": "1e89d8aea28d75bc7b1f15576ee7893cfe1eadcb", "dirty": false, - "recorded_at": "2026-09-09T01:48:28Z", + "recorded_at": "2026-09-09T01:58:04Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 34.5 + "elapsed_seconds": 34.1 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 29.4 + "elapsed_seconds": 30.4 }, { "id": "M02", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.0 + "elapsed_seconds": 28.1 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 27.9 + "elapsed_seconds": 28.4 }, { "id": "M05", @@ -60,7 +60,7 @@ "catchers": [ "stage1::raw-rc-retained" ], - "elapsed_seconds": 28.9 + "elapsed_seconds": 28.8 }, { "id": "M06", @@ -68,7 +68,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 27.9 + "elapsed_seconds": 28.6 }, { "id": "M07", @@ -76,7 +76,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 28.1 }, { "id": "M08", @@ -84,7 +84,7 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.2 + "elapsed_seconds": 28.1 }, { "id": "M09", @@ -92,16 +92,17 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 28.0 }, { "id": "M10", "outcome": "caught", "catchers": [ + "stage1::compare-no-substitution", "stage1::compare-same-input", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.5 + "elapsed_seconds": 28.4 }, { "id": "M11", @@ -109,7 +110,7 @@ "catchers": [ "stage1::exec-failure-is-5" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 28.3 }, { "id": "M12", @@ -117,7 +118,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 28.3 }, { "id": "M13", @@ -125,15 +126,16 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 28.9 }, { "id": "M14", "outcome": "caught", "catchers": [ + "stage1::compare-no-substitution", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.0 + "elapsed_seconds": 28.3 }, { "id": "M15", @@ -141,7 +143,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 27.7 }, { "id": "M16", @@ -149,7 +151,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 29.0 + "elapsed_seconds": 28.5 } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md new file mode 100644 index 00000000..a9650683 --- /dev/null +++ b/docs/generated/p022-stage1-mutations.md @@ -0,0 +1,43 @@ + + +# P-022 step 8 (#262) Stage 1 — mutation campaigns + +Stage 1 makes the Rust core SELECTABLE by the launcher while Python stays the default and the reference. Every mutation below is a plausible MISREADING of that contract rather than a syntactic accident: the default moved because the cutover was read as already decided; Python resolved for every engine because the old unconditional resolution looked harmless; 70 admitted to the verdict set because both engines document it; an unexpected child status propagated as itself because that looked like faithfulness; the shell falling back to Python because answering the user looked helpful. Each would pass a reviewer who had read the stage's summary instead of its rulings. Every mutation edits a **production** launcher surface (P-022 discipline 2), and the single layer REBUILDS the launcher before testing it — a mutated `.cs` file is otherwise invisible to controls that drive a compiled binary — and runs every control for every mutation with no fail-fast (discipline 3), under `OWEN_STAGE1_REQUIRE=1` so a control that could not run is a failure rather than a silently shrinking denominator. The counts are derived from the recorded run by `scripts/mutate_campaign.summarize()`, never typed. + +## Stage 1 — the launcher's `--engine` contract: the default, the candidate locator, the Rust child status and the compare result contract + +Campaign `p022-stage1-1` — #262 Stage 1 — the launcher's engine-selection contract: that Python stays the default, that an explicitly selected Rust core actually runs (and needs no Python), that a Rust failure is never a Python success, that an unexpected child status becomes public exit 5 with the raw status retained, that an unusable OWEN_RUST_CORE is a configuration error rather than a fallback, and that compare extracts once, feeds both engines the same bytes, and refuses to answer when they disagree or when either fails. Every mutation is a plausible MISREADING of that contract rather than a syntactic accident: each one would pass a reviewer who had read the stage's summary instead of its rulings. Every mutation edits a PRODUCTION launcher surface, and every declared layer runs for every mutation (no fail-fast). + +Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-1.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. + +| measure | value | +|--------------------------------------------------|---| +| recorded at commit | `1e89d8aea28d75bc7b1f15576ee7893cfe1eadcb` | +| layers run (every one, for every mutation) | `stage1` | +| mutations | 16 | +| caught | 16 | +| survived | 0 | +| compile-error (no evidence either way) | 0 | +| invalid-mutation | 0 | +| runner-error | 0 | +| caught without every expected catcher | none | +| honesty control `M00` (unmutated tree must pass) | survived — as required | + +| id | rule | mutation | outcome | caught by | +|---|---|---|---|---| +| M01 | default-is-python | the Stage-1 default engine is Rust — the cutover read as already decided, instead of Python remaining default until Gate G3 | caught | `stage1::default-stays-python` | +| M02 | rust-run-needs-no-python | Python is resolved for every engine — the old unconditional resolution kept 'just in case', which silently re-imposes a Python dependency on a Rust-only run | caught | `stage1::rust-actually-runs-rust` | +| M03 | 70-is-not-a-verdict | 70 is treated as a legal engine result — the shared internal-error code mistaken for part of the verdict contract because both engines document it | caught | `stage1::rc70-is-not-a-verdict`
`stage1::rust-failure-no-fallback` | +| M04 | unexpected-rc-maps-to-5 | an unexpected Rust child status passes through as itself — 'propagate the child's exit code' read as faithfulness rather than as leaking a meaningless number to the caller | caught | `stage1::raw-rc-retained`
`stage1::rc70-is-not-a-verdict`
`stage1::rust-failure-no-fallback`
`stage1::unexpected-rc-maps-to-5` | +| M05 | raw-rc-retained | the raw child status is dropped from the report — the human-readable cause already names the number, so the typed carrier looks redundant | caught | `stage1::raw-rc-retained` | +| M06 | bad-locator-is-2-not-5 | an unusable OWEN_RUST_CORE is an internal error — a failure to start the engine read as Owen's own bug rather than the caller's configuration | caught | `stage1::bad-locator-is-2` | +| M07 | bad-locator-is-2-not-3 | an unusable OWEN_RUST_CORE reuses exit 3 — 'no usable engine runtime' read as the same class as 'no usable Python', which it is not | caught | `stage1::bad-locator-is-2` | +| M08 | no-fallback-in-the-shell | the shell falls back to Python when the Rust core produces no verdict — 'be helpful, still give the user an answer', which is precisely the silent fallback every ruling forbids | caught | `stage1::rust-failure-no-fallback` | +| M09 | shell-locator-is-2-not-3 | the shell reports an unusable OWEN_RUST_CORE as exit 3 — the Python-specific 'no usable runtime' code borrowed for the Rust candidate | caught | `stage1::bad-locator-is-2` | +| M10 | divergence-is-5 | a divergence exposes the reference's result — 'Python is still the reference, so trust it' read as a licence to answer while the two engines disagree | caught | `stage1::compare-no-substitution`
`stage1::compare-same-input`
`stage1::divergence-is-5` | +| M11 | exec-failure-is-not-agreement | the execution-failure check is skipped — comparing the results first and treating a crashed engine as just another difference, which loses the distinction D4.1 draws between (b) and (c) | caught | `stage1::exec-failure-is-5` | +| M12 | zero-document-compare-fails | a document with no analysable unit is compared anyway — 'both engines agreed' read as a result rather than as a zero denominator | caught | `stage1::compare-zero-document` | +| M13 | candidate-identity-recorded | the evidence records a placeholder candidate digest — the path already names the binary, so hashing it looks like belt-and-braces | caught | `stage1::candidate-identity` | +| M14 | shell-compare-checks-stdout | the shell compare stops comparing stdout — the exit code read as the whole of 'the public result', dropping the bytes the user actually sees | caught | `stage1::compare-no-substitution`
`stage1::divergence-is-5` | +| M15 | shell-divergence-is-5 | the shell reports a divergence as exit 1 — the 'something is wrong' tier reached for, when in public Owen 1 already means findings | caught | `stage1::divergence-is-5` | +| M16 | shell-zero-document-fails | the shell's zero-document guard is dropped — an empty document read as a legitimately clean agreement | caught | `stage1::compare-zero-document` | diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index b5a51cd6..9a17087e 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -510,7 +510,8 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None lines = tally.read_text(encoding="utf-8").splitlines() if tally.exists() else [] extractions = [ln for ln in lines if "OwnSharp.Extractor" in ln] if len(extractions) != 1: - fail(once_check, f"compare invoked the extractor {len(extractions)} times, expected exactly 1") + fail(once_check, + f"compare invoked the extractor {len(extractions)} times, expected exactly 1") else: ok(once_check, "compare extracted exactly once") @@ -547,7 +548,8 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None fail(same_check, "the compare run wrote no evidence to attest the capture digest") return try: - attested = (json.loads(evidence.read_text(encoding="utf-8")).get("input") or {}).get("sha256") + recorded = json.loads(evidence.read_text(encoding="utf-8")) + attested = (recorded.get("input") or {}).get("sha256") except (OSError, json.JSONDecodeError) as exc: fail(same_check, f"the compare evidence is unreadable: {exc}") return From 154387851076606b30326003a87a2da224aaed6d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:08:50 +0000 Subject: [PATCH 06/22] evidence(stage1): record the mutation campaign and its generated counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 16/16 mutations caught, 0 survived, 0 compile-error, 0 invalid, 0 runner-error, every expected catcher hit, and the M00 honesty control survived the unmutated tree — recorded on a clean tree at 063ac628d8d4, the commit this PR ships. The counts in docs/generated/p022-stage1-mutations.md are projected from the recorded run by scripts/render_checkpoint_status.py and never typed; tests/test_checkpoint_status.py fails while the fragment is stale. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 36 ++++++++++++------------- docs/generated/p022-stage1-mutations.md | 2 +- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index 2514dc9b..501023f0 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -4,9 +4,9 @@ "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", - "source_commit": "1e89d8aea28d75bc7b1f15576ee7893cfe1eadcb", + "source_commit": "063ac628d8d417fa12c4b02f364b29c4324395bb", "dirty": false, - "recorded_at": "2026-09-09T01:58:04Z", + "recorded_at": "2026-09-09T02:07:56Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 34.1 + "elapsed_seconds": 35.5 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 30.4 + "elapsed_seconds": 30.8 }, { "id": "M02", @@ -32,7 +32,7 @@ "catchers": [ "stage1::rust-actually-runs-rust" ], - "elapsed_seconds": 26.8 + "elapsed_seconds": 28.1 }, { "id": "M03", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 30.6 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 28.4 + "elapsed_seconds": 28.6 }, { "id": "M05", @@ -60,7 +60,7 @@ "catchers": [ "stage1::raw-rc-retained" ], - "elapsed_seconds": 28.8 + "elapsed_seconds": 28.1 }, { "id": "M06", @@ -68,7 +68,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 27.7 }, { "id": "M07", @@ -84,7 +84,7 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 28.4 }, { "id": "M09", @@ -92,7 +92,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.0 + "elapsed_seconds": 28.6 }, { "id": "M10", @@ -102,7 +102,7 @@ "stage1::compare-same-input", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.4 + "elapsed_seconds": 27.8 }, { "id": "M11", @@ -110,7 +110,7 @@ "catchers": [ "stage1::exec-failure-is-5" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 28.5 }, { "id": "M12", @@ -118,7 +118,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 28.9 }, { "id": "M13", @@ -126,7 +126,7 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 28.9 + "elapsed_seconds": 29.0 }, { "id": "M14", @@ -135,7 +135,7 @@ "stage1::compare-no-substitution", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 28.5 }, { "id": "M15", @@ -143,7 +143,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 27.7 + "elapsed_seconds": 28.2 }, { "id": "M16", @@ -151,7 +151,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.5 + "elapsed_seconds": 28.9 } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index a9650683..c97f76b2 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -12,7 +12,7 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 | measure | value | |--------------------------------------------------|---| -| recorded at commit | `1e89d8aea28d75bc7b1f15576ee7893cfe1eadcb` | +| recorded at commit | `063ac628d8d417fa12c4b02f364b29c4324395bb` | | layers run (every one, for every mutation) | `stage1` | | mutations | 16 | | caught | 16 | From 69c1b501e58efa9253e8ad999ab3dbb57edbc959 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:19:37 +0000 Subject: [PATCH 07/22] fix(stage1): make the engine contract hold on Windows, not only on Linux MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows leg of the new job failed on its first run. Four distinct causes, three of them real defects rather than test noise. 1. A LATENT PRE-EXISTING DEFECT, newly exposed. Every own-check.sh invocation on Windows exited 1 before reaching any of its own logic. The repository has no .gitattributes, so a Windows checkout (core.autocrlf=true by default) rewrites scripts/own-check.sh to CRLF, and bash cannot run a script whose every line ends in a stray carriage return. No job had ever run own-check.sh on a Windows runner — the three that use it are ubuntu-only — so this sat latent until this stage asked for the shell surface on both platforms. Fixed with a deliberately NARROW `*.sh text eol=lf`: this repository has byte-sensitive evidence whose exact bytes are the contract, and a blanket `* text=auto` would put those at the mercy of a checkout setting. #260's wider .gitattributes tail is untouched. 2. A REAL GAP IN D3.1's SEAM. Windows has no execute bit, so a non-executable candidate passed the locator's checks, was spawned, and died as exit 5. D3.1 draws the line at STARTING: "bad locator / cannot select the candidate -> rc 2; candidate spawned -> rc 5". A candidate that never started is on the rc-2 side, so a spawn failure is now caught and reported as the configuration error it is — on the launcher (a typed RustCoreNotStartedException) and in the shell (bash's 126/127). On Windows this is the ONLY point at which a non-runnable candidate can be detected. 3. MY OWN DESIGN COLLIDING WITH ITSELF. The stub-driven controls were Unix-only, and OWEN_STAGE1_REQUIRE=1 turns every skip into a failure — so the flag that exists to stop a denominator shrinking was failing controls that could not exist on the platform. The two most important of them no longer need a stub at all: the unexpected-child-status and raw-status controls now force an uncatchable death in the REAL candidate through #261's existing OWN_CLI_FAULT_ABORT, which runs on both platforms and proves a property of the binary the launcher actually spawns. The remainder are declared `not applicable` on Windows — printed and counted separately, never silently skipped — because a synthetic candidate needs a shebang. 4. A CROSS-RUNTIME CONVENTION, not a defect. Python's subprocess reports a signal-killed child as the negative signal number (-6); .NET and every shell report 128+signal (134). The launcher records what .NET observed, so the control translates into that convention rather than asking the launcher to adopt Python's. Also: the extraction-count control now separates an INSTRUMENT failure from a contract failure. If its PATH shim never intercepts `dotnet` while the compare itself succeeds, it reports "not applicable" instead of accusing the launcher of an extraction count the measurement could not see. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .gitattributes | 16 +++ frontend/roslyn/OwnSharp.Cli/CheckCommand.cs | 17 ++- frontend/roslyn/OwnSharp.Cli/CompareMode.cs | 9 +- frontend/roslyn/OwnSharp.Cli/EngineRunner.cs | 27 ++++- scripts/own-check.sh | 17 +++ tests/test_stage1_engine.py | 104 +++++++++++++++---- 6 files changed, 162 insertions(+), 28 deletions(-) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..b4e3ed8d --- /dev/null +++ b/.gitattributes @@ -0,0 +1,16 @@ +# Shell scripts must arrive with LF endings on every platform. +# +# `scripts/own-check.sh` is executed by git-bash on Windows (#262 Stage 1's +# engine controls run it there). A checkout with the Windows default +# `core.autocrlf=true` would rewrite it to CRLF, and bash cannot run a script +# whose every line ends in a stray carriage return — it fails immediately, with +# an exit code that looks like the script's own. No job had ever run this +# script on a Windows runner before, so the defect was latent rather than new. +# +# Deliberately NARROW. This repository has byte-sensitive evidence (fixtures, +# goldens, recorded campaign results) whose exact bytes are the contract, and a +# blanket `* text=auto` would put those bytes at the mercy of a checkout +# setting. #260 carries "a researched .gitattributes for byte-sensitive +# evidence" as its own hygiene tail; this rule covers executable shell scripts +# only and takes no position on that wider question. +*.sh text eol=lf diff --git a/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs b/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs index e239c73c..e2a7caf1 100644 --- a/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs +++ b/frontend/roslyn/OwnSharp.Cli/CheckCommand.cs @@ -180,9 +180,20 @@ await RunExtractorAsync(paths, factsPath, legacy, stats, bodyThrowEdges) // branches and nowhere else. if (engine == Engine.Rust) { - var rustOutcome = await EngineRunner - .RunRustAsync(rustCore!, factsPath, format, severity, capture: false) - .ConfigureAwait(false); + EngineOutcome rustOutcome; + try + { + rustOutcome = await EngineRunner + .RunRustAsync(rustCore!, factsPath, format, severity, capture: false) + .ConfigureAwait(false); + } + catch (EngineRunner.RustCoreNotStartedException ex) + { + // D3.1's seam: the candidate could not be STARTED, so this + // is still "cannot select the candidate" — exit 2, not 5. + Console.Error.WriteLine(ex.Message); + return RustCoreLocator.ExitCode; + } return MapRustChildStatus(rustOutcome.Rc, args, failOnFinding); } diff --git a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs index a9d78756..e41ed1b5 100644 --- a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs +++ b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs @@ -121,10 +121,13 @@ public static async Task RunAsync( .RunRustAsync(rust, rustInput, format, severity, capture: true) .ConfigureAwait(false); } - catch (Exception ex) when (ex is InvalidOperationException or IOException) + catch (EngineRunner.RustCoreNotStartedException ex) { - return Fail(args, rust, py, null, - $"the Rust candidate could not be run: {ex.Message}", childExitCode: null); + // Same seam as `--engine rust`: a candidate that never started + // is a configuration error (exit 2), not a compare execution + // failure (exit 5). The compare never happened. + Console.Error.WriteLine(ex.Message); + return RustCoreLocator.ExitCode; } // --- D4.1 (c): execution failure ------------------------------ diff --git a/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs b/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs index c5ba4c06..58453947 100644 --- a/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs +++ b/frontend/roslyn/OwnSharp.Cli/EngineRunner.cs @@ -71,6 +71,17 @@ public static async Task RunPythonAsync( return await RunAsync(psi, capture, "the Python core").ConfigureAwait(false); } + /// + /// Thrown when the resolved candidate could not be STARTED. That is still + /// the locator's side of D3.1's seam: "bad locator / cannot select the + /// candidate → rc 2; candidate spawned → a legal engine result is the + /// normal contract, an unexpected child result → rc 5". A file that exists + /// but is not a runnable image is a configuration mistake, not Owen + /// failing internally — and on Windows it is the ONLY way to detect a + /// non-executable candidate at all, since there is no execute bit to test. + /// + public sealed class RustCoreNotStartedException(string message) : Exception(message); + /// /// Stage 2, the Rust candidate: the production executable `own-cli ownir`. /// @@ -98,7 +109,21 @@ public static async Task RunRustAsync( psi.EnvironmentVariables["OWNLANG_DEBUG"] = "1"; } - return await RunAsync(psi, capture, "the Rust core").ConfigureAwait(false); + try + { + return await RunAsync(psi, capture, "the Rust core").ConfigureAwait(false); + } + catch (Exception ex) when (ex is System.ComponentModel.Win32Exception + or IOException or InvalidOperationException or UnauthorizedAccessException) + { + // The candidate never started. Windows has no execute bit, so this + // is where a non-executable candidate is caught there; on Unix the + // locator's mode check catches it first and this is the backstop. + throw new RustCoreNotStartedException( + $"owen check: the candidate `own-cli` binary could not be started: " + + $"'{core.Path}' ({ex.Message}). Set {RustCoreLocator.EnvVar} to a runnable " + + $"`own-cli` executable. Owen did not fall back to Python."); + } } /// Start the child and collect its result. Capturing drains both diff --git a/scripts/own-check.sh b/scripts/own-check.sh index e4ede189..d1564d4c 100755 --- a/scripts/own-check.sh +++ b/scripts/own-check.sh @@ -215,6 +215,16 @@ case "$engine" in "$rust_core" ownir "$facts" --format "$format" --severity "$severity" rc=$? set -e + # The candidate could not be STARTED at all: bash reports 126 for "found + # but not executable" and 127 for "not found". That is still the locator's + # side of D3.1's seam — cannot select the candidate — so it is a + # configuration error (2), not an internal failure (5). On Windows, where + # there is no execute bit to test up front, this is the ONLY place a + # non-runnable candidate can be caught. + if [[ "$rc" -eq 126 || "$rc" -eq 127 ]]; then + echo "own-check: the candidate \`own-cli\` binary could not be started: '$rust_core' (exit $rc). Set OWEN_RUST_CORE to a runnable \`own-cli\` executable. Owen did not fall back to Python." >&2 + exit 2 + fi # 0/1/2 are verdicts and pass through. Anything else — 70, a panic, a # signal death, an arbitrary 42 — is NOT a verdict: it takes the public # internal-error path (5) with the raw status named on stderr, and it never @@ -284,6 +294,13 @@ ZERODOC # failure — checked before divergence, because two results are only # comparable once both exist. No engine's answer substitutes for the # other's failure. + # A candidate that never started is a configuration error, not a compare + # execution failure: the compare did not happen. + if [[ "$rs_rc" -eq 126 || "$rs_rc" -eq 127 ]]; then + echo "own-check: the candidate \`own-cli\` binary could not be started: '$rust_core' (exit $rs_rc). Set OWEN_RUST_CORE to a runnable \`own-cli\` executable. Owen did not fall back to Python." >&2 + exit 2 + fi + py_legal=0; rs_legal=0 [[ "$py_rc" -eq 0 || "$py_rc" -eq 1 || "$py_rc" -eq 2 ]] && py_legal=1 [[ "$rs_rc" -eq 0 || "$rs_rc" -eq 1 || "$rs_rc" -eq 2 ]] && rs_legal=1 diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index 9a17087e..d5773dea 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -98,6 +98,7 @@ _FAILURES: list[tuple[str, str]] = [] _PASSES: list[str] = [] _SKIPS: list[tuple[str, str]] = [] +_NOT_APPLICABLE: list[tuple[str, str]] = [] def fail(check: str, detail: str) -> None: @@ -121,6 +122,22 @@ def skip(check: str, why: str) -> None: print(f"skip[{check}]: {why}") +def not_applicable(check: str, why: str) -> None: + """A control that CANNOT exist on this platform, as opposed to one that + could not run here. + + The difference is not bookkeeping. `skip()` means the toolchain was + missing, and under OWEN_STAGE1_REQUIRE that is a failure because the CI job + exists to supply it. This means the control is not a question this platform + can be asked — a synthetic candidate binary needs a shebang, and Windows + has none — so requiring it would only produce a red job that no amount of + correct code could turn green. It is still printed, and still counted + separately, so a control cannot quietly disappear behind it. + """ + _NOT_APPLICABLE.append((check, why)) + print(f"n/a[{check}]: {why}") + + # --- toolchain ------------------------------------------------------------- @@ -375,31 +392,61 @@ def control_rc70_is_not_a_verdict(sample: Path) -> None: def control_unexpected_rc_and_raw_retention(sample: Path, tmp: Path) -> None: """An unexpected child status maps to public 5, and the RAW status is - retained in the diagnostic report's typed `child_exit_code` (D5). Two - controls, one forced condition: the mapping and the retention fail - independently and are reported independently.""" + retained in the diagnostic report's typed `child_exit_code` (D5). + + The unexpected status is forced with #261's own `OWN_CLI_FAULT_ABORT` — an + uncatchable termination in the REAL candidate — rather than with a stub + that merely exits with a chosen number. That buys two things: the control + runs on Windows as well as Linux (a shebang stub does not), and what it + proves is a property of the binary the launcher will actually spawn. The + exact raw status is whatever the OS reports for an aborted process, which + differs by platform and is deliberately NOT contracted (#261's ruling); the + control measures it first and then requires the report to carry that same + value. + """ map_check, keep_check = "unexpected-rc-maps-to-5", "raw-rc-retained" - if os.name == "nt": - skip(map_check, "stub candidate is Unix-only") - skip(keep_check, "stub candidate is Unix-only") + fault = rust_fault_core() + if fault is None: + skip(map_check, "no OWEN_STAGE1_RUST_FAULT") + skip(keep_check, "no OWEN_STAGE1_RUST_FAULT") + return + + # What does an aborted candidate actually exit with here? Measured, not + # assumed — and if it lands inside the legal set the control cannot speak. + probe = subprocess.run([fault, "ownir", "--format", "human", str(tmp / "no-such-facts.json")], + capture_output=True, check=False, + env={**os.environ, "OWN_CLI_FAULT_ABORT": "1"}) + # Two runtimes, two conventions for the same event: Python's subprocess + # reports a signal-killed child as the NEGATIVE signal number (-6 for + # SIGABRT), while .NET's Process.ExitCode — and every shell — reports + # 128 + signal (134). Neither is wrong; they describe the same death. The + # launcher records what .NET observed, so the expectation is translated + # into that convention rather than the launcher being asked to adopt + # Python's. On Windows there is no signal encoding and the code is already + # what both sides see. + raw = probe.returncode if probe.returncode >= 0 else 128 - probe.returncode + if raw in (0, 1, 2): + skip(map_check, f"the forced abort produced a legal engine exit ({raw})") + skip(keep_check, f"the forced abort produced a legal engine exit ({raw})") return - stub = write_stub(tmp / "rc42-core", 42, stderr="forced unexpected status\n") + report = Path.home() / ".owen/diag/last-failure.json" if report.exists(): report.unlink() r = run_owen(["--engine", "rust", "--format", "human", str(sample)], - env={"OWEN_RUST_CORE": str(stub)}) + env={"OWEN_RUST_CORE": fault, "OWN_CLI_FAULT_ABORT": "1"}) if r is None: skip(map_check, "no built launcher/dotnet") skip(keep_check, "no built launcher/dotnet") return - if r.returncode == 42: - fail(map_check, "an unexpected child rc 42 escaped as the public exit 42") + if r.returncode == raw: + fail(map_check, f"an unexpected child rc {raw} escaped as the public exit {raw}") elif r.returncode != 5: - fail(map_check, f"an unexpected child rc 42 surfaced as {r.returncode}, expected public 5") + fail(map_check, + f"an unexpected child rc {raw} surfaced as {r.returncode}, expected public 5") else: - ok(map_check, "an unexpected child rc 42 maps to public exit 5") + ok(map_check, f"an unexpected child rc {raw} maps to public exit 5") if not report.exists(): fail(keep_check, "no diagnostic report was written for an unexpected child status") @@ -412,16 +459,16 @@ def control_unexpected_rc_and_raw_retention(sample: Path, tmp: Path) -> None: if "child_exit_code" not in data: fail(keep_check, "the report has no `child_exit_code` field (D5)") return - if data["child_exit_code"] != 42: - fail(keep_check, f"`child_exit_code` is {data['child_exit_code']!r}, expected the raw 42") - return if not isinstance(data["child_exit_code"], int): - fail(keep_check, "`child_exit_code` is not a typed integer") + fail(keep_check, f"`child_exit_code` is {data['child_exit_code']!r}, not a typed integer") + return + if data["child_exit_code"] != raw: + fail(keep_check, f"`child_exit_code` is {data['child_exit_code']}, expected the raw {raw}") return if data.get("schema") != 2: fail(keep_check, f"the report schema is {data.get('schema')!r}, expected 2 (D5 bumped it)") return - ok(keep_check, "the raw 42 is retained as a typed child_exit_code, schema 2") + ok(keep_check, f"the raw {raw} is retained as a typed child_exit_code, schema 2") def control_no_selector_in_own_cli() -> None: @@ -509,7 +556,16 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None lines = tally.read_text(encoding="utf-8").splitlines() if tally.exists() else [] extractions = [ln for ln in lines if "OwnSharp.Extractor" in ln] - if len(extractions) != 1: + if not lines: + # The shim never intercepted anything, yet the compare itself + # succeeded — so `dotnet` was resolved past it (git-bash on Windows + # prefers `dotnet.exe` to an extensionless script). That is the + # INSTRUMENT failing, not the contract: reporting "0 extractions" + # would accuse the launcher of a fault the measurement cannot see. + not_applicable(once_check, + "the PATH shim did not intercept `dotnet` on this platform, so extraction " + "count could not be measured (the compare itself succeeded)") + elif len(extractions) != 1: fail(once_check, f"compare invoked the extractor {len(extractions)} times, expected exactly 1") else: @@ -521,7 +577,7 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None # digest the launcher attests in its evidence. If compare ever fed the two # engines different bytes, these two values part company. if os.name == "nt": - skip(same_check, "recording stub is Unix-only") + not_applicable(same_check, "needs a recording stub candidate; Unix-only (see above)") return seen = tmp / "candidate-saw.sha256" recorder = tmp / "recording-core" @@ -616,7 +672,10 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: sub_check = "compare-no-substitution" if os.name == "nt": for c in (div_check, exec_check, sub_check): - skip(c, "stub candidate is Unix-only") + not_applicable(c, "needs a synthetic candidate with chosen output; a shebang stub " + "is Unix-only, and the launcher cannot spawn a .cmd. The C# and " + "bash logic under test is platform-independent and is measured on " + "the Linux leg") return if not have_dotnet(): for c in (div_check, exec_check, sub_check): @@ -777,7 +836,10 @@ def run() -> int: print() print(f"stage-1 engine controls: {len(_PASSES)} passed, " - f"{len(_FAILURES)} failed, {len(_SKIPS)} skipped") + f"{len(_FAILURES)} failed, {len(_SKIPS)} skipped, " + f"{len(_NOT_APPLICABLE)} not applicable on this platform") + for name, why in _NOT_APPLICABLE: + print(f" n/a {name}: {why}") if _SKIPS: print(" skipped (set OWEN_STAGE1_REQUIRE=1 to make these failures):") for name, why in _SKIPS: From 2dac238d81e682b683b17ef5ac9f342ed534087c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:26:19 +0000 Subject: [PATCH 08/22] fix(stage1): normalise Windows child status; surface the child's stderr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-ups from the second Windows run, which improved to 6 passing. The raw-status control compared 3221226505 with -1073740791 and called them different. They are the same 32-bit value: Python reports a Windows status unsigned, .NET reports the same bits as a signed int32. The earlier translation covered Unix signal numbers only. All three conventions are now mapped into the one the launcher actually records: Unix, Python subprocess -6 the negative signal number Unix, .NET / any shell 134 128 + signal Windows, Python 3221226505 0xC0000409, unsigned Windows, .NET -1073740791 the same bits, signed int32 These are measured facts about the runtimes, not fudge factors — the launcher is not being asked to adopt Python's spelling of a status. The second failure I cannot yet explain: own-check.sh exits 1 on Windows after passing its locator check, producing no verdict and no findings. It is not the CRLF defect (that is fixed — the locator control now passes there through this same script), and I cannot reproduce it on this machine. Rather than guess at it across CI cycles, the two controls that hit it now include the script's own stderr and stdout in their failure message. A control that reports only "expected 5, got 1" keeps the reason to itself, and on a platform the author cannot reproduce that reason is the whole diagnosis. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- tests/test_stage1_engine.py | 53 ++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index d5773dea..bddfe392 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -188,6 +188,23 @@ def run_owen(args: list[str], env: dict[str, str] | None = None capture_output=True, env=e, cwd=str(ROOT), check=False) +def tail(r: subprocess.CompletedProcess[bytes], limit: int = 500) -> str: + """The child's own words, for a failure message. + + A control that says only "expected 2, got 1" hands the reader a number and + keeps the reason to itself — and when the failure happens on a platform the + author cannot reproduce, that reason is the whole diagnosis. + """ + err = r.stderr.decode("utf-8", "replace").strip() + out = r.stdout.decode("utf-8", "replace").strip() + parts = [] + if err: + parts.append(f"stderr: …{err[-limit:]}") + if out: + parts.append(f"stdout: …{out[-limit:]}") + return " | ".join(parts) or "(both streams empty)" + + def write_stub(path: Path, exit_code: int, stdout: str = "", stderr: str = "") -> Path: """A candidate that exits with a chosen code. Unix only — the real fault-injection binary covers both platforms for the cases it can force.""" @@ -346,7 +363,10 @@ def control_rust_failure_no_fallback(sample: Path) -> None: env={"OWEN_RUST_CORE": fault, "OWN_CLI_FAULT_PANIC": "1"}) if r2.returncode in (0, 1): problems.append(f"own-check.sh: a forced Rust failure exited {r2.returncode} — a verdict " - "was produced despite the engine failing") + f"was produced despite the engine failing [{tail(r2)}]") + elif r2.returncode != 5: + problems.append(f"own-check.sh: a forced Rust failure exited {r2.returncode}, expected " + f"public 5 [{tail(r2)}]") if b"OWN001" in r2.stdout: problems.append("own-check.sh: a forced Rust failure still produced findings — Python " "answered for Rust") @@ -416,15 +436,23 @@ def control_unexpected_rc_and_raw_retention(sample: Path, tmp: Path) -> None: probe = subprocess.run([fault, "ownir", "--format", "human", str(tmp / "no-such-facts.json")], capture_output=True, check=False, env={**os.environ, "OWN_CLI_FAULT_ABORT": "1"}) - # Two runtimes, two conventions for the same event: Python's subprocess - # reports a signal-killed child as the NEGATIVE signal number (-6 for - # SIGABRT), while .NET's Process.ExitCode — and every shell — reports - # 128 + signal (134). Neither is wrong; they describe the same death. The - # launcher records what .NET observed, so the expectation is translated - # into that convention rather than the launcher being asked to adopt - # Python's. On Windows there is no signal encoding and the code is already - # what both sides see. - raw = probe.returncode if probe.returncode >= 0 else 128 - probe.returncode + # Three conventions for the same death, none of them wrong: + # + # Unix, Python subprocess -6 the negative signal number + # Unix, .NET / any shell 134 128 + signal + # Windows, Python 3221226505 0xC0000409, unsigned + # Windows, .NET -1073740791 the same bits, signed int32 + # + # The launcher records what .NET observed, so the expectation is + # translated into .NET's convention rather than the launcher being asked + # to adopt Python's. Both translations are measured facts about the + # runtimes, not fudge factors: the signal form is mapped to 128+signal, + # and an unsigned 32-bit status is reinterpreted as signed. + raw = probe.returncode + if raw < 0 and raw >= -128: # a Unix signal number + raw = 128 - raw + elif raw > 0x7FFFFFFF: # an unsigned Windows status + raw -= 0x100000000 if raw in (0, 1, 2): skip(map_check, f"the forced abort produced a legal engine exit ({raw})") skip(keep_check, f"the forced abort produced a legal engine exit ({raw})") @@ -652,9 +680,10 @@ def control_compare_zero_document(tmp: Path) -> None: return if r.returncode in (0, 1): problems.append(f"{where}: a zero-document compare exited {r.returncode} — it " - "passed instead of failing") + f"passed instead of failing [{tail(r)}]") elif r.returncode != 5: - problems.append(f"{where}: a zero-document compare exited {r.returncode}, expected 5") + problems.append(f"{where}: a zero-document compare exited {r.returncode}, expected 5 " + f"[{tail(r)}]") elif "nothing to analyse" not in merged: problems.append(f"{where}: a zero-document compare failed without saying why") if problems: From ceba906ac0e713ea763ca3c49f0fe28370986b99 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:31:00 +0000 Subject: [PATCH 09/22] fix(stage1): run own-check.sh through git-bash, not WSL's bash.exe stub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The instrumentation added in the previous commit paid for itself immediately. The Windows failure was never in the launcher: the controls' own stderr, once printed, turned out to be UTF-16 text reading "You can resolve this by installing a distribution... Use 'wsl.exe --list --online'". `subprocess.run(["bash", ...])` on a Windows runner resolves `bash` to C:\Windows\System32\bash.exe — the WSL LAUNCHER, not a shell. With no distribution installed it prints that notice and exits 1, which arrives at a control as a plausible-looking "own-check.sh exited 1" and is nothing of the kind. Every own-check.sh-driven control on Windows was measuring WSL's absence. The harness now names Git for Windows' bash explicitly and refuses anything under System32. This is a harness concern rather than a product one: a Windows user runs own-check.sh from a git-bash prompt, where `bash` already is the right one, and the script itself needed no change. Worth recording as method, not just as a fix: three CI rounds of guessing would not have found this, and one round of printing the child's own words did. A control that reports "expected 5, got 1" and keeps the reason to itself is a control that cannot be debugged on a platform its author cannot run. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- tests/test_stage1_engine.py | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index bddfe392..c7aea2af 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -166,13 +166,41 @@ def have_dotnet() -> bool: return shutil.which("dotnet") is not None +def bash_exe() -> str: + """The bash that can actually run `own-check.sh`. + + On a Windows runner `bash` on PATH is `C:\\Windows\\System32\\bash.exe` — + the WSL launcher, not a shell. With no distribution installed it prints + "You can resolve this by installing a distribution..." to stderr in UTF-16 + and exits 1, which arrives here as a plausible-looking script failure and + is nothing of the kind. Git for Windows ships the bash that own-check.sh is + written for, so it is named explicitly and System32 is refused outright. + + This is a HARNESS concern, not a product one: a Windows user running + own-check.sh does so from a git-bash prompt, where `bash` is already the + right one. + """ + if os.name != "nt": + return "bash" + candidates = [ + os.environ.get("SHELL"), + r"C:\Program Files\Git\bin\bash.exe", + r"C:\Program Files\Git\usr\bin\bash.exe", + shutil.which("bash"), + ] + for cand in candidates: + if cand and "system32" not in cand.lower() and Path(cand).is_file(): + return cand + return "bash" + + def run_own_check(args: list[str], env: dict[str, str] | None = None, cwd: str | None = None) -> subprocess.CompletedProcess[bytes]: """own-check.sh, with raw bytes: the compare contract is about bytes.""" e = dict(os.environ) e.update(env or {}) return subprocess.run( - ["bash", str(ROOT / "scripts/own-check.sh"), *args], + [bash_exe(), str(ROOT / "scripts/own-check.sh"), *args], capture_output=True, env=e, cwd=cwd or str(ROOT), check=False) @@ -261,7 +289,7 @@ def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: e.pop("OWEN_RUST_CORE", None) if kind == "shell": r = subprocess.run( - ["bash", str(ROOT / "scripts/own-check.sh"), + [bash_exe(), str(ROOT / "scripts/own-check.sh"), "--engine", engine, "--", str(sample)], capture_output=True, env=e, cwd=str(ROOT), check=False) else: From 4b45469fe3564e5cdf0565ed851f3e6a7420a42f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:37:09 +0000 Subject: [PATCH 10/22] fix(stage1): a Windows compare divergence is the declared A/B/C change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows leg reached 9 passing and then failed on something that is not a defect at all: `--engine compare` diverges there, so two controls that required agreement failed. Measured rather than reasoned: the divergence appears on PURE-ASCII output. The Python reference writes CRLF where the Rust core writes LF, so the two engines' bytes differ on every Windows run — before encoding enters the picture at all. That is #262's Windows A/B/C behaviour change, and requiring agreement on that platform would be requiring exactly the parity #262 explicitly does not claim. The controls now treat a divergence as a legitimate compare outcome: the extraction count is measurable whether the compare agreed, diverged, or failed, because extraction happens before any of them. This also corrects my own documentation, which was too narrow. Both the launcher README and CompareMode's contract comment said Windows compare would diverge "for non-ASCII output". The measurement says every run. The docs now say so, and separate the two halves of the difference: line endings on all output, encoding (cp1252 vs canonical UTF-8, and the reference's occasional UnicodeEncodeError) on non-ASCII. Worth noting what caught this: comparing raw BYTES rather than decoded text. A string comparison through one .NET encoding would have normalised the line endings away and reported agreement — hiding a declared difference behind a green control. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- frontend/roslyn/OwnSharp.Cli/CompareMode.cs | 10 +++++++--- frontend/roslyn/OwnSharp.Cli/README.md | 17 +++++++++++----- tests/test_stage1_engine.py | 22 ++++++++++++++++----- 3 files changed, 36 insertions(+), 13 deletions(-) diff --git a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs index e41ed1b5..21284cda 100644 --- a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs +++ b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs @@ -32,9 +32,13 @@ namespace OwnSharp.Cli; /// /// Known difference, not a defect. On native Windows the Python /// reference encodes piped output as cp1252 with CRLF while the Rust candidate -/// emits canonical UTF-8 (#262's Windows A/B/C). Compare will therefore report -/// a real divergence there for non-ASCII output. That is the declared -/// behaviour change being visible, which is the point of measuring bytes. +/// emits canonical UTF-8 with LF (#262's Windows A/B/C). Measured on a Windows +/// runner, that makes compare diverge on EVERY run, not only on non-ASCII +/// output: the line endings alone differ, so pure-ASCII findings already +/// disagree byte for byte. This is the declared behaviour change being +/// visible, which is exactly the point of comparing bytes rather than decoded +/// text — a string comparison through one .NET encoding would have hidden +/// it. /// internal static class CompareMode { diff --git a/frontend/roslyn/OwnSharp.Cli/README.md b/frontend/roslyn/OwnSharp.Cli/README.md index ad29ce3b..daef9ca2 100644 --- a/frontend/roslyn/OwnSharp.Cli/README.md +++ b/frontend/roslyn/OwnSharp.Cli/README.md @@ -89,11 +89,18 @@ exit (`5`) with the raw status kept in the diagnostic report's `compare` is a development/CI seam for the migration, **not yet a promised public feature**. When the engines disagree — or when either fails — owen exits `5` with reproduction evidence rather than picking a winner: a reference -and a candidate that disagree mean owen cannot honestly emit one answer. On -native Windows the Python reference emits cp1252/CRLF where the Rust core -emits canonical UTF-8, so `compare` will report a real divergence there for -non-ASCII output; that is #262's declared Windows behaviour change being -visible, not a defect. +and a candidate that disagree mean owen cannot honestly emit one answer. + +**On native Windows `compare` diverges on every run**, and that is #262's +declared Windows A/B/C behaviour change being visible rather than a defect. +Measured on a Windows runner: the Python reference writes **CRLF** line +endings where the Rust core writes LF, so the two engines' bytes differ even +for pure-ASCII output — the encoding half (cp1252 vs canonical UTF-8, and the +reference's occasional `UnicodeEncodeError`) is the further difference on +non-ASCII output. Canonical/Linux reference parity is claimed and Rust +cross-platform byte portability is claimed; native-Windows Python byte parity +is explicitly **not**. Use `compare` against the Linux reference; on Windows, +read a divergence as the recorded difference, not as a finding. Rollback is explicit: select `--engine python` (or simply stop passing `--engine`). Nothing about Stage 1 moves the public default. diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index c7aea2af..c7026168 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -604,10 +604,19 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None env = {"OWEN_RUST_CORE": core, "PATH": f"{shim_dir}{os.pathsep}{os.environ.get('PATH','')}"} r = run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], env=env) - if r.returncode not in (0, 1): - fail(once_check, f"compare over a healthy sample exited {r.returncode}: " - f"{r.stderr.decode('utf-8', 'replace')[-400:]}") - fail(same_check, "compare did not reach agreement, so same-input could not be observed") + # 0/1 = agreement, 5 = divergence or execution failure. ALL THREE mean the + # compare ran, and the extraction happened before any of them, so the count + # below is measurable in every case. + # + # On native Windows a divergence here is EXPECTED, not a defect: the Python + # reference writes CRLF where the Rust core writes LF, so the two engines' + # bytes differ on every run — #262's declared Windows A/B/C behaviour + # change, measured. Requiring agreement on that platform would be requiring + # the parity #262 explicitly does not claim. + if r.returncode not in (0, 1, 5): + fail(once_check, f"compare over a healthy sample exited {r.returncode}, so it never " + f"reached extraction [{tail(r)}]") + fail(same_check, "compare never ran, so same-input could not be observed") return lines = tally.read_text(encoding="utf-8").splitlines() if tally.exists() else [] @@ -633,7 +642,10 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None # digest the launcher attests in its evidence. If compare ever fed the two # engines different bytes, these two values part company. if os.name == "nt": - not_applicable(same_check, "needs a recording stub candidate; Unix-only (see above)") + not_applicable(same_check, "needs a recording stub candidate with a shebang; Unix-only. " + "The launcher code that materialises and verifies the two " + "engine inputs is platform-independent and is measured on the " + "Linux leg") return seen = tmp / "candidate-saw.sha256" recorder = tmp / "recording-core" From e7170cdfe0dd250e494e97b179cd58b1d8aaadd5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:42:19 +0000 Subject: [PATCH 11/22] fix(stage1): a non-executable candidate is not a state git-bash can be shown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last Windows failure was a fixture artifact, not a product defect. The "non-executable" candidate is a mode-644 file carrying a shebang, and git-bash runs those quite happily — Windows has no execute bit — so own-check.sh correctly proceeded, the stub ran, and the control saw exit 0 where it wanted exit 2. Nothing in the launcher or the script was wrong. The case is now declared inapplicable to the SHELL surface on Windows, and the control counts its own denominator instead of reporting a fixed 20, so it cannot claim coverage it did not have. The same case remains real and still runs for the LAUNCHER on Windows: there the candidate is spawned as a Windows process, fails to start, and the D3.1 spawn-seam backstop added earlier turns that into exit 2 — which is why that half was already passing in this run while the shell half was not. The two surfaces genuinely differ here because the platform gives them different questions to answer, and the controls now say so rather than averaging over it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- tests/test_stage1_engine.py | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index c7026168..5b772f51 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -297,6 +297,15 @@ def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: ["dotnet", str(launcher_dll()), "check", "--engine", engine, str(sample)], capture_output=True, env=e, cwd=str(ROOT), check=False) where = f"{surface_name}/{engine}/{name}" + if name == "non-executable" and kind == "shell" and os.name == "nt": + # Windows has no execute bit, and git-bash will happily run + # a mode-644 file that carries a shebang — so "a + # non-executable candidate" is not a state that can be + # constructed for the shell surface there. It is a real + # case for the LAUNCHER on Windows, where the candidate is + # spawned as a Windows process and fails to start; that + # half still runs and still expects exit 2. + continue if r.returncode != 2: problems.append(f"{where}: exit {r.returncode}, expected 2 " "(not 3 — that is Python-specific; not 5 — that is an " @@ -311,7 +320,17 @@ def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: if problems: fail(check, "; ".join(problems)) else: - ok(check, f"{len(cases) * 2 * len(surfaces)} invalid-locator cases all exit 2, no fallback") + # Counted, not assumed: the Windows shell surface skips one case that + # cannot exist there, and a control that reported a fixed number would + # be claiming coverage it did not have. + total = len(cases) * 2 * len(surfaces) + if os.name == "nt": + total -= 2 # the two engines' non-executable case, shell surface + not_applicable(check + "/shell-non-executable", + "git-bash runs a mode-644 file with a shebang, so a non-executable " + "candidate cannot be constructed for the shell surface on Windows; " + "the launcher half of this case does run") + ok(check, f"{total} invalid-locator cases all exit 2, no fallback") def control_default_stays_python(sample: Path) -> None: From 0f51868df54041b8a8ed2af1c8b0ce41838009c8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 02:49:18 +0000 Subject: [PATCH 12/22] fix(stage1): own-check.ps1 writes stderr text, not PowerShell ErrorRecords MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows leg got all the way to its last step: the controls passed with no FAIL lines and the step printed "own-check.ps1 engine contract OK", meaning every assertion held. It failed anyway, for two reasons that have nothing to do with the engine contract. A real production wart. own-check.ps1 reported its diagnostics with Write-Error, which emits an ErrorRecord rather than text. A caller running with $ErrorActionPreference = 'Stop' — which is exactly how GitHub's pwsh shell runs — turns that into a TERMINATING error in the caller, even though the script had handled the condition and exited with a deliberate code. A command-line tool's diagnostics belong on stderr as text, the way own-check.sh has always written them. Reproduced locally under a strict caller before and after: the bad-locator case now prints its line, exits 2, and the caller carries on. A harness bug beside it. GitHub's pwsh wrapper appends `exit $LASTEXITCODE`, and this step's last real command exits 2 deliberately — the case it exists to require. Without an explicit `exit 0` the step inherited the exit code of its own passing assertion. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .github/workflows/ci.yml | 6 ++++++ scripts/own-check.ps1 | 27 +++++++++++++-------------- 2 files changed, 19 insertions(+), 14 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 08d0276c..8d57c814 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -218,6 +218,12 @@ jobs: & ./scripts/own-check.ps1 -Engine rust -Format human -- $sample if ($LASTEXITCODE -ne 2) { throw "own-check.ps1 with a bad OWEN_RUST_CORE expected exit 2, got $LASTEXITCODE" } Write-Host "own-check.ps1 engine contract OK" + # The assertions above ARE this step's verdict, so say so + # explicitly. GitHub's pwsh wrapper appends `exit $LASTEXITCODE`, + # and the last command here exits 2 ON PURPOSE — without this the + # step would inherit the exit code of a case it was written to + # require. + exit 0 # P-022 step 7a (#260) — COMPARE MODE over the committed corpus: the FAST half # of #260's test matrix, and one leg of it. The five pinned OSS repositories, diff --git a/scripts/own-check.ps1 b/scripts/own-check.ps1 index c77b79be..2a6b7400 100644 --- a/scripts/own-check.ps1 +++ b/scripts/own-check.ps1 @@ -114,9 +114,9 @@ if ($Engine -eq "rust" -or $Engine -eq "compare") { # Windows has no execute bit: an existing regular file is accepted here # and a genuinely broken image fails at spawn, which is the other side # of the D3.1 seam and already maps to the internal-error path. - Write-Error -Message ("own-check: --engine $Engine needs the candidate ``own-cli`` binary, but " + + [Console]::Error.WriteLine(("own-check: --engine $Engine needs the candidate ``own-cli`` binary, but " + "OWEN_RUST_CORE $problem. Set OWEN_RUST_CORE to the absolute path of the ``own-cli`` " + - "executable to run. Owen did not fall back to Python.") -ErrorAction Continue + "executable to run. Owen did not fall back to Python.")) exit 2 } } @@ -159,8 +159,8 @@ try { # and takes the public internal-error path (5) with the raw child # status named. It never runs Python instead. if ($rc -ne 0 -and $rc -ne 1 -and $rc -ne 2) { - Write-Error -Message ("own-check: the Rust analysis core exited $rc, which is not a " + - "verdict (raw child status: $rc). Owen did not fall back to Python.") -ErrorAction Continue + [Console]::Error.WriteLine(("own-check: the Rust analysis core exited $rc, which is not a " + + "verdict (raw child status: $rc). Owen did not fall back to Python.")) exit 5 } } @@ -187,9 +187,9 @@ try { if ($null -ne $v -and $null -ne $v.Value -and @($v.Value).Count -gt 0) { $hasUnit = $true; break } } if (-not $hasUnit) { - Write-Error -Message ("own-check: --engine compare: the captured OwnIR contains nothing to " + + [Console]::Error.WriteLine(("own-check: --engine compare: the captured OwnIR contains nothing to " + "analyse — a compare over zero documents proves nothing and is a failure, not an " + - "agreement.") -ErrorAction Continue + "agreement.")) exit 5 } } @@ -201,9 +201,9 @@ try { $pyInSha = (Get-FileHash -LiteralPath $pyIn -Algorithm SHA256).Hash.ToLowerInvariant() $rsInSha = (Get-FileHash -LiteralPath $rsIn -Algorithm SHA256).Hash.ToLowerInvariant() if ($pyInSha -ne $captureSha -or $rsInSha -ne $captureSha) { - Write-Error -Message ("own-check: --engine compare: the two engine inputs are not byte-identical " + + [Console]::Error.WriteLine(("own-check: --engine compare: the two engine inputs are not byte-identical " + "to the single capture (capture $captureSha, python $pyInSha, rust $rsInSha) — the " + - "same-input invariant failed, so no comparison may be reported.") -ErrorAction Continue + "same-input invariant failed, so no comparison may be reported.")) exit 5 } @@ -228,11 +228,10 @@ try { $pyLegal = ($pyRc -eq 0 -or $pyRc -eq 1 -or $pyRc -eq 2) $rsLegal = ($rsRc -eq 0 -or $rsRc -eq 1 -or $rsRc -eq 2) if (-not $pyLegal -or -not $rsLegal) { - Write-Error -Message ("own-check: --engine compare: compare execution failure (python exit " + + [Console]::Error.WriteLine(("own-check: --engine compare: compare execution failure (python exit " + "$pyRc, rust exit $rsRc). No engine's result was substituted for the other's failure. " + - "Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts in $cmpDir") ` - -ErrorAction Continue - if (-not $rsLegal) { Write-Error -Message "own-check: raw Rust child status: $rsRc" -ErrorAction Continue } + "Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts in $cmpDir")) + if (-not $rsLegal) { [Console]::Error.WriteLine("own-check: raw Rust child status: $rsRc") } exit 5 } @@ -246,10 +245,10 @@ try { if ($pyOutH -ne $rsOutH) { $diverged += "stdout" } if ($pyErrH -ne $rsErrH) { $diverged += "stderr" } if ($diverged.Count -gt 0) { - Write-Error -Message ("own-check: --engine compare: engine divergence — the reference and the " + + [Console]::Error.WriteLine(("own-check: --engine compare: engine divergence — the reference and the " + "candidate disagree on " + ($diverged -join ", ") + ". Neither verdict is exposed as " + "authoritative. Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts " + - "in $cmpDir") -ErrorAction Continue + "in $cmpDir")) # Keep the artifacts for reproduction rather than deleting them. $keep = $true exit 5 From 2d6e43ad0b08489762d8ba072c851802ad302e34 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 03:07:46 +0000 Subject: [PATCH 13/22] evidence(stage1): re-record the campaign against the tree CI proved green MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 16/16 caught, 0 survived, 0 compile-error, 0 invalid, 0 runner-error, every expected catcher hit, and the M00 honesty control survived the unmutated tree — recorded on a clean tree at 0f51868df540, which is the commit whose CI run was fully green on both platforms. The earlier recording (063ac628d8d4) is superseded and was never shippable: the Windows work changed CheckCommand.cs, CompareMode.cs, EngineRunner.cs, own-check.sh and own-check.ps1 after it was taken, so it named a tree that no longer existed. Evidence that describes a tree nobody can check out is not evidence, whatever its counts say. Counts in docs/generated/p022-stage1-mutations.md are projected from this run by scripts/render_checkpoint_status.py and never typed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 36 ++++++++++++------------- docs/generated/p022-stage1-mutations.md | 2 +- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index 501023f0..a182c69a 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -4,9 +4,9 @@ "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", - "source_commit": "063ac628d8d417fa12c4b02f364b29c4324395bb", + "source_commit": "0f51868df54041b8a8ed2af1c8b0ce41838009c8", "dirty": false, - "recorded_at": "2026-09-09T02:07:56Z", + "recorded_at": "2026-09-09T03:07:20Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 35.5 + "elapsed_seconds": 48.9 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 30.8 + "elapsed_seconds": 30.1 }, { "id": "M02", @@ -32,7 +32,7 @@ "catchers": [ "stage1::rust-actually-runs-rust" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 27.1 }, { "id": "M03", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 30.6 + "elapsed_seconds": 28.7 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 27.8 }, { "id": "M05", @@ -68,7 +68,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 27.7 + "elapsed_seconds": 28.0 }, { "id": "M07", @@ -76,7 +76,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 27.6 }, { "id": "M08", @@ -84,7 +84,7 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.4 + "elapsed_seconds": 27.6 }, { "id": "M09", @@ -92,7 +92,7 @@ "catchers": [ "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.6 + "elapsed_seconds": 27.5 }, { "id": "M10", @@ -102,7 +102,7 @@ "stage1::compare-same-input", "stage1::divergence-is-5" ], - "elapsed_seconds": 27.8 + "elapsed_seconds": 28.0 }, { "id": "M11", @@ -110,7 +110,7 @@ "catchers": [ "stage1::exec-failure-is-5" ], - "elapsed_seconds": 28.5 + "elapsed_seconds": 27.7 }, { "id": "M12", @@ -118,7 +118,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.9 + "elapsed_seconds": 28.3 }, { "id": "M13", @@ -126,7 +126,7 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 29.0 + "elapsed_seconds": 28.1 }, { "id": "M14", @@ -135,7 +135,7 @@ "stage1::compare-no-substitution", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.5 + "elapsed_seconds": 28.9 }, { "id": "M15", @@ -143,7 +143,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 28.2 + "elapsed_seconds": 28.4 }, { "id": "M16", @@ -151,7 +151,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.9 + "elapsed_seconds": 28.5 } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index c97f76b2..a7cee27b 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -12,7 +12,7 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 | measure | value | |--------------------------------------------------|---| -| recorded at commit | `063ac628d8d417fa12c4b02f364b29c4324395bb` | +| recorded at commit | `0f51868df54041b8a8ed2af1c8b0ce41838009c8` | | layers run (every one, for every mutation) | `stage1` | | mutations | 16 | | caught | 16 | From 7ca3655fe5b363d5decf75e183d94ea01ab3bf0f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:08:43 +0000 Subject: [PATCH 14/22] fix(stage1): five review defects in identity, classification and PS semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified each against primary source before touching it; all five were real. D1 — a relative OWEN_RUST_CORE was accepted by all three independent implementations. `Path.GetFullPath` reads as "it resolves the path for me", and it does: against whatever directory Owen happened to run in, so the same configuration would select different binaries from different places. That is the ambient resolution D3 exists to forbid. Now rejected up front, exit 2, naming the requirement. The shell test accepts the absolute forms this surface actually receives — the MSYS `/d/a/...` CI passes, `C:\...`, `C:/...`, UNC — because a bare `/*` glob would turn a correct Windows configuration into a usage error; it converts between none of them, as D3 ratified a locator and not a path-translation policy. action.yml is not a fourth implementation: it delegates, and the control proves the forwarding. D2 — the compare verdict was inferred as `childExitCode is null && py is not null && rs is not null ? "divergence" : "execution-failure"`. `child_exit_code` is the D5 RUST-child carrier, so a Python-only bad exit left it null with both outcomes present and stamped "divergence" onto evidence whose own diagnostic beside it said "execution failure". The classification is now passed by the call site, which is the only place that knows. The external vocabulary is unchanged — agreement / divergence / execution-failure — because a new serialized value is a contract the project would owe support for forever. D3 — own-check.ps1 never mapped a spawn failure to exit 2, and its comment asserted the opposite of D3.1 ("maps to the internal-error path"). A candidate that never started is on the locator's side of the seam. Both engine paths now catch it, and the comment says what the code does. D4 — ps1 agreement replayed through `Get-Content -Raw | Write-Output`, a decode-and-re-encode, and emitted no stderr at all. It now replays the reference's raw bytes on both streams, as C# and own-check.sh do. Implemented for the contract, not for today's statistics: CRLF-vs-LF makes agreement rare on Windows now, but a replay that is wrong only when it finally runs is worse than none. D5 — ps1 named $cmpDir as reproduction evidence and then deleted it in `finally`. Pointing a reader at a path and shredding it on the way out is worse than naming nothing. The directory now survives an execution failure, as it already did a divergence. A SIXTH defect, found while proving D4 and fixed with it: `Start-Process -RedirectStandardOutput` is not byte-faithful. Measured on one input, the Python reference wrote 211 bytes and the redirected file held 210 — a blank line silently dropped. Compare claims the engines' public BYTES are identical, so a lossy capture of the reference can manufacture a divergence that does not exist or hide one that does; an agreement reached over a corrupted capture is not an agreement. ps1 now drains both pipes as byte streams, concurrently, the way the C# launcher does. Fixing only the replay would have left the mechanism broken underneath it. Evidence. `tests/helpers/stage1_stub.rs` is a committed, controllable native candidate compiled with plain `rustc` — no cargo crate, so #261's crate-edge DAG gate is untouched. It replaces the shebang stubs, which were Unix-only and had forced four compare controls to be declared not-applicable on Windows: that gap is why three PowerShell defects survived a 16/16 campaign. All seventeen shared controls now run on both platforms with zero N/A, and `tests/test_stage1_ps1.py` adds the PowerShell surface to the adversarial set. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .github/workflows/ci.yml | 93 ++++- docs/evidence/p022-stage1-1.json | 37 +- docs/evidence/p022-stage1-ps1.json | 68 ++++ frontend/roslyn/OwnSharp.Cli/CompareMode.cs | 42 ++- .../roslyn/OwnSharp.Cli/RustCoreLocator.cs | 17 + scripts/own-check.ps1 | 140 ++++++- scripts/own-check.sh | 22 ++ scripts/render_checkpoint_status.py | 3 + tests/helpers/stage1_stub.rs | 107 ++++++ tests/test_stage1_engine.py | 261 +++++++++++-- tests/test_stage1_ps1.py | 349 ++++++++++++++++++ 11 files changed, 1070 insertions(+), 69 deletions(-) create mode 100644 docs/evidence/p022-stage1-ps1.json create mode 100644 tests/helpers/stage1_stub.rs create mode 100644 tests/test_stage1_ps1.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8d57c814..64656f4b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -154,6 +154,14 @@ jobs: - name: Build the fault-injection own-cli (forced failure modes) working-directory: rust run: cargo build -p own-cli --release --features fault-injection --target-dir target-fault + # The controllable native candidate the compare controls need. Built + # with plain `rustc`, deliberately NOT as a cargo workspace member, so it + # cannot move the crate-edge DAG that #261's gate pins. + - name: Build the Stage-1 stub candidate + run: | + ext="" + if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi + rustc -O tests/helpers/stage1_stub.rs -o "$RUNNER_TEMP/stage1-stub$ext" - name: Build the owen launcher run: dotnet build frontend/roslyn/OwnSharp.Cli/OwnSharp.Cli.csproj -c Release --nologo # The vendored Python core is a PACK-time payload, so a plain build does @@ -164,7 +172,7 @@ jobs: out=frontend/roslyn/OwnSharp.Cli/bin/Release/net8.0 mkdir -p "$out/ownlang-core/ownlang" cp ownlang/*.py "$out/ownlang-core/ownlang/" - - name: Stage-1 engine controls (all fifteen, no fail-fast) + - name: Stage-1 engine controls (no fail-fast) env: OWEN_STAGE1_REQUIRE: "1" run: | @@ -172,8 +180,22 @@ jobs: if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli$ext" export OWEN_STAGE1_RUST_FAULT="$PWD/rust/target-fault/release/own-cli$ext" + export OWEN_STAGE1_STUB="$RUNNER_TEMP/stage1-stub$ext" export OWEN_STAGE1_LAUNCHER_DLL="$PWD/frontend/roslyn/OwnSharp.Cli/bin/Release/net8.0/ownsharp.dll" python tests/test_stage1_engine.py + # The PowerShell surface's own controls. They run on BOTH legs — the + # logic is platform-neutral and a Linux run catches regressions early — + # but only the Windows leg is evidence for a PowerShell-targeted + # mutation, which is what the stage1-ps1-mutations job below settles. + - name: Stage-1 PowerShell controls + env: + OWEN_STAGE1_REQUIRE: "1" + run: | + ext="" + if [ "${{ matrix.os }}" = "windows-latest" ]; then ext=".exe"; fi + export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli$ext" + export OWEN_STAGE1_STUB="$RUNNER_TEMP/stage1-stub$ext" + python tests/test_stage1_ps1.py # The explicit Rust-selected run on this platform, through the shell # launcher, recorded as its own step so the evidence names the surface # and the platform rather than being inferred from a green job. @@ -225,6 +247,75 @@ jobs: # require. exit 0 + # P-022 step 8 (#262) Stage 1 — the WINDOWS-NATIVE mutation leg. + # + # A mutation whose target is scripts/own-check.ps1 is only `caught` when a + # Windows PowerShell catcher observes the mutant and fails. Running those + # mutants on Linux would execute the mutated PowerShell under a different + # runtime, and the Windows-specific halves — the spawn seam above all — + # cannot be settled there at all: a mutant that runs where its control is + # weakest is decorative, and proves nothing about the surface it edits. + # + # So this campaign runs here, on Windows, and this job is the gate. It fails + # unless every mutation is caught with its expected catcher and the + # honesty control survives the unmutated tree. + stage1-ps1-mutations: + name: own-check.ps1 mutation campaign (Windows-native) + runs-on: windows-latest + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + - uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4 + with: + dotnet-version: "8.0.x" + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.13" + - uses: dtolnay/rust-toolchain@fa04a1451ff1842e2626ccb99004d0195b455a88 # master, 2026-07-10 + with: + toolchain: stable + - name: Build the production own-cli candidate + working-directory: rust + run: cargo build -p own-cli --release + - name: Build the Stage-1 stub candidate + run: rustc -O tests/helpers/stage1_stub.rs -o "$RUNNER_TEMP/stage1-stub.exe" + - name: Run the PowerShell mutation campaign + env: + OWEN_STAGE1_REQUIRE: "1" + run: | + export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli.exe" + export OWEN_STAGE1_STUB="$RUNNER_TEMP/stage1-stub.exe" + python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-ps1.json --run + # The recorded run, printed in full so its provenance and per-mutation + # catchers can be read off this job rather than taken on trust — and so + # the committed record can be reproduced from a named CI run, the way + # #260's sweep record is. + - name: Print the recorded result + if: always() + run: cat docs/evidence/p022-stage1-ps1.result.json + - name: Assert every mutation was caught by a Windows catcher + run: | + python - <<'PY' + import json, sys + d = json.load(open("docs/evidence/p022-stage1-ps1.result.json", encoding="utf-8")) + defn = json.load(open("docs/evidence/p022-stage1-ps1.json", encoding="utf-8")) + exp = {m["id"]: set(m["expected_catchers"]) for m in defn["mutations"]} + problems = [] + if d["control"]["outcome"] != "survived": + problems.append("the honesty control did not survive the unmutated tree") + for m in d["mutations"]: + if m["outcome"] != "caught": + problems.append(f"{m['id']}: {m['outcome']}") + elif not exp[m["id"]] <= set(m["catchers"]): + problems.append(f"{m['id']}: expected catchers missed ({m['catchers']})") + elif not any(c.startswith("ps1::") for c in m["catchers"]): + problems.append(f"{m['id']}: no PowerShell catcher observed it") + print("\n".join(problems) if problems else "every ps1 mutation caught by a Windows catcher") + sys.exit(1 if problems else 0) + PY + # P-022 step 7a (#260) — COMPARE MODE over the committed corpus: the FAST half # of #260's test matrix, and one leg of it. The five pinned OSS repositories, # the large-solution controls and the examples tree are the scheduled/manual diff --git a/docs/evidence/p022-stage1-1.json b/docs/evidence/p022-stage1-1.json index 6112fc03..2a238fd6 100644 --- a/docs/evidence/p022-stage1-1.json +++ b/docs/evidence/p022-stage1-1.json @@ -126,8 +126,8 @@ "rule": "divergence-is-5", "description": "a divergence exposes the reference's result — 'Python is still the reference, so trust it' read as a licence to answer while the two engines disagree", "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", - "pattern": " return Fail\\(args, rust, py, rs,\\n \\$\\\"engine divergence", - "replacement": " await ReplayAsync(py).ConfigureAwait(false);\n return failOnFinding ? py.Rc : (py.Rc >= 2 ? py.Rc : 0);\n#pragma warning disable CS0162\n return Fail(args, rust, py, rs,\n $\"engine divergence", + "pattern": " return Fail\\(args, rust, py, rs, Divergence,\\n \\$\\\"engine divergence", + "replacement": " await ReplayAsync(py).ConfigureAwait(false);\n return failOnFinding ? py.Rc : (py.Rc >= 2 ? py.Rc : 0);\n#pragma warning disable CS0162\n return Fail(args, rust, py, rs, Divergence,\n $\"engine divergence", "expected_catchers": [ "stage1::divergence-is-5", "stage1::compare-no-substitution" @@ -198,6 +198,39 @@ "expected_catchers": [ "stage1::compare-zero-document" ] + }, + { + "id": "M17", + "rule": "locator-must-be-absolute", + "description": "the launcher accepts a relative OWEN_RUST_CORE — `GetFullPath` reads as 'it resolves the path for me', which it does: against whatever directory Owen happened to run in", + "target": "frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs", + "pattern": " if \\(!Path\\.IsPathFullyQualified\\(raw\\)\\)", + "replacement": " if (false)", + "expected_catchers": [ + "stage1::absolute-locator-only" + ] + }, + { + "id": "M18", + "rule": "shell-locator-must-be-absolute", + "description": "the shell accepts a relative OWEN_RUST_CORE — the -f/-x tests look like they answer 'is this a usable binary', and they do, for whatever the current directory made of the path", + "target": "scripts/own-check.sh", + "pattern": " elif \\[\\[ \\\"\\$is_absolute\\\" -eq 0 \\]\\]; then", + "replacement": " elif false; then", + "expected_catchers": [ + "stage1::absolute-locator-only" + ] + }, + { + "id": "M19", + "rule": "compare-verdict-is-stated", + "description": "the compare verdict is inferred from the Rust-child field again — `child_exit_code is null` reads as 'no engine crashed', but it is only ever about the RUST child, so a Python-only failure is stamped 'divergence'", + "target": "frontend/roslyn/OwnSharp.Cli/CompareMode.cs", + "pattern": " verdict: verdict, diagnostic: diagnostic, childExitCode: childExitCode\\);", + "replacement": " verdict: childExitCode is null && py is not null && rs is not null\n ? \"divergence\"\n : \"execution-failure\",\n diagnostic: diagnostic, childExitCode: childExitCode);", + "expected_catchers": [ + "stage1::compare-failure-classified" + ] } ] } diff --git a/docs/evidence/p022-stage1-ps1.json b/docs/evidence/p022-stage1-ps1.json new file mode 100644 index 00000000..8060e9c5 --- /dev/null +++ b/docs/evidence/p022-stage1-ps1.json @@ -0,0 +1,68 @@ +{ + "schema": 1, + "comment": "GENERATED-BY-HAND definition; the RESULT beside it is recorded by scripts/mutate_campaign.py --run on a WINDOWS runner and the counts are derived from it, never typed.", + "campaign": "p022-stage1-ps1", + "description": "#262 Stage 1 — the PowerShell launcher surface. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Every mutation here targets own-check.ps1 itself, and every one is a plausible MISREADING rather than a syntactic accident — Test-Path answering a question D3 did not ask, a failed spawn read as the engine blowing up, `Get-Content | Write-Output` read as an echo, cleanup read as tidiness.", + "layers": [ + { + "id": "ps1", + "cwd": ".", + "parser": "python-fail", + "command": [ + "python", + "tests/test_stage1_ps1.py" + ] + } + ], + "layers_comment": "WINDOWS-NATIVE BY CONTRACT. A mutation whose target is scripts/own-check.ps1 is only `caught` when a Windows PowerShell catcher observes the mutant and fails; this campaign therefore runs on a windows-latest runner (.github/workflows/ci.yml, job `stage1-ps1-mutations`). Running it on Linux would execute the mutated PowerShell under a different runtime and could not settle the Windows-specific halves — the spawn seam above all — so a Linux result is not evidence for these mutants and is never recorded as one.", + "control": { + "id": "M00", + "description": "harness-honesty control: no mutation at all, which must report zero failing layers" + }, + "mutations": [ + { + "id": "P01", + "rule": "ps1-locator-must-be-absolute", + "description": "own-check.ps1 accepts a relative OWEN_RUST_CORE — Test-Path says the file is there, which is true and not the question D3 asks", + "target": "scripts/own-check.ps1", + "pattern": " elseif \\(-not \\[System\\.IO\\.Path\\]::IsPathFullyQualified\\(\\$rustCore\\)\\) \\{", + "replacement": " elseif ($false) {", + "expected_catchers": [ + "ps1::ps1-absolute-locator" + ] + }, + { + "id": "P02", + "rule": "ps1-not-started-is-configuration", + "description": "own-check.ps1 reports a candidate that never started as an internal failure — 'the engine blew up' read as Owen's bug rather than the caller's configuration, which is the exact side of D3.1's seam the old comment got backwards", + "target": "scripts/own-check.ps1", + "pattern": " exit 2\n \\}\n # 0/1/2 are verdicts and pass through", + "replacement": " exit 5\n }\n # 0/1/2 are verdicts and pass through", + "expected_catchers": [ + "ps1::ps1-not-started-is-2" + ] + }, + { + "id": "P03", + "rule": "ps1-agreement-replays-raw-bytes", + "description": "own-check.ps1 replays agreement through the object pipeline again, stdout only — `Get-Content | Write-Output` looks like an echo and is a decode-and-re-encode that also drops stderr", + "target": "scripts/own-check.ps1", + "pattern": " \\$outBytes = \\[System\\.IO\\.File\\]::ReadAllBytes\\(\\(Join-Path \\$cmpDir \"python\\.out\"\\)\\)\n \\$errBytes = \\[System\\.IO\\.File\\]::ReadAllBytes\\(\\(Join-Path \\$cmpDir \"python\\.err\"\\)\\)", + "replacement": " Get-Content -LiteralPath (Join-Path $cmpDir \"python.out\") -Raw -ErrorAction SilentlyContinue | Write-Output\n $outBytes = @(); $errBytes = @()", + "expected_catchers": [ + "ps1::ps1-agreement-replays" + ] + }, + { + "id": "P04", + "rule": "ps1-failure-evidence-survives", + "description": "own-check.ps1 deletes the reproduction directory it just named — cleanup reads as tidiness, and the message that pointed at it is left describing something that no longer exists", + "target": "scripts/own-check.ps1", + "pattern": " # reader at a path and then deleting it on the way out is worse\n # than not naming one at all\\.\n \\$keep = \\$true", + "replacement": " # reader at a path and then deleting it on the way out is worse\n # than not naming one at all.\n $keep = $false", + "expected_catchers": [ + "ps1::ps1-failure-evidence" + ] + } + ] +} diff --git a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs index 21284cda..14d9afdf 100644 --- a/frontend/roslyn/OwnSharp.Cli/CompareMode.cs +++ b/frontend/roslyn/OwnSharp.Cli/CompareMode.cs @@ -46,6 +46,14 @@ internal static class CompareMode /// public internal-error path. public const int ExitCode = CrashReport.ExitCode; + /// The evidence's external verdict vocabulary. Three + /// values, written down once so a call site cannot invent a fourth: what a + /// consumer reads has to stay stable, and every case here is one of + /// "they agreed", "they disagreed", or "one of them did not answer". + private const string Agreement = "agreement"; + private const string Divergence = "divergence"; + private const string ExecutionFailure = "execution-failure"; + /// Run both engines over one capture and apply D4.1. /// The public exit code: the reference's own on agreement, else 5. public static async Task RunAsync( @@ -67,7 +75,7 @@ public static async Task RunAsync( } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) { - return Fail(args, rust, null, null, + return Fail(args, rust, null, null, ExecutionFailure, $"could not capture the extracted OwnIR: {ex.Message}", childExitCode: null); } @@ -81,7 +89,7 @@ public static async Task RunAsync( // zero denominator wearing a passing grade. if (!HasAnalyzableUnit(captured, out var why)) { - return Fail(args, rust, null, null, + return Fail(args, rust, null, null, ExecutionFailure, $"the captured OwnIR contains nothing to analyse ({why}) — a compare over " + "zero documents proves nothing and is a failure, not an agreement", childExitCode: null); @@ -99,7 +107,7 @@ public static async Task RunAsync( var rustSha = Sha256Hex(await File.ReadAllBytesAsync(rustInput).ConfigureAwait(false)); if (pythonSha != capturedSha || rustSha != capturedSha) { - return Fail(args, rust, null, null, + return Fail(args, rust, null, null, ExecutionFailure, "the two engine inputs are not byte-identical to the single capture " + $"(capture {capturedSha}, python {pythonSha}, rust {rustSha}) — the " + "same-input invariant failed, so no comparison may be reported", @@ -116,7 +124,7 @@ public static async Task RunAsync( } catch (Exception ex) when (ex is InvalidOperationException or IOException) { - return Fail(args, rust, null, null, + return Fail(args, rust, null, null, ExecutionFailure, $"the Python reference could not be run: {ex.Message}", childExitCode: null); } try @@ -150,7 +158,7 @@ public static async Task RunAsync( : !pyLegal ? $"the Python reference failed (exit {py.Rc})" : $"the Rust candidate failed (exit {rs.Rc})"; - return Fail(args, rust, py, rs, + return Fail(args, rust, py, rs, ExecutionFailure, $"compare execution failure — {offender}. No engine's result was " + "substituted for the other's failure.", childExit); @@ -168,7 +176,7 @@ public static async Task RunAsync( sameOut ? null : $"stdout ({py.Stdout.Length} vs {rs.Stdout.Length} bytes)", sameErr ? null : $"stderr ({py.Stderr.Length} vs {rs.Stderr.Length} bytes)", }.Where(x => x is not null)); - return Fail(args, rust, py, rs, + return Fail(args, rust, py, rs, Divergence, $"engine divergence — the reference and the candidate disagree on {what}. " + "Neither verdict is exposed as authoritative: Owen cannot honestly emit " + "one answer when its reference and its candidate disagree.", @@ -180,7 +188,7 @@ public static async Task RunAsync( // the user sees exactly what a `--engine python` run would have // produced — byte for byte, replayed undecoded. await ReplayAsync(py).ConfigureAwait(false); - WriteEvidence(args, rust, capturedSha, py, rs, verdict: "agreement", + WriteEvidence(args, rust, capturedSha, py, rs, verdict: Agreement, diagnostic: null, childExitCode: null); return failOnFinding ? py.Rc : (py.Rc >= 2 ? py.Rc : 0); } @@ -190,8 +198,21 @@ public static async Task RunAsync( TryDelete(rustInput); } + // The classification is PASSED IN, never inferred. It used to be + // derived as `childExitCode is null && py is not null && rs is not + // null ? "divergence" : "execution-failure"`, which reads the D5 Rust + // child carrier as if it were a classifier: a Python-only bad exit + // leaves childExitCode null with both outcomes present, so the + // evidence said "divergence" while the diagnostic beside it said + // "execution failure". Only the call site knows which case it is, so + // only the call site says. + // + // `verdict` keeps its existing external vocabulary — agreement / + // divergence / execution-failure. A finer distinction belongs in the + // human-readable `diagnostic`, not in a new serialized value the + // project would then owe support for. int Fail(string[] a, RustCore core, EngineOutcome? py, EngineOutcome? rs, - string diagnostic, int? childExitCode) + string verdict, string diagnostic, int? childExitCode) { Console.Error.WriteLine($"owen: --engine compare: {diagnostic}"); // The reproduction line is on STDERR, not only inside the evidence @@ -204,10 +225,7 @@ int Fail(string[] a, RustCore core, EngineOutcome? py, EngineOutcome? rs, $" Reproduction — input sha256 {(capturedSha.Length > 0 ? capturedSha : "(no capture)")}, " + $"candidate {core.Path} (sha256 {core.Sha256})"); var path = WriteEvidence(a, core, capturedSha, py, rs, - verdict: childExitCode is null && py is not null && rs is not null - ? "divergence" - : "execution-failure", - diagnostic: diagnostic, childExitCode: childExitCode); + verdict: verdict, diagnostic: diagnostic, childExitCode: childExitCode); if (path is not null) { Console.Error.WriteLine($" Reproduction evidence: {path}"); diff --git a/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs index f37373f0..e740f829 100644 --- a/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs +++ b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs @@ -67,6 +67,23 @@ public static RustCore Resolve() // A directory that exists still fails the File.Exists test below, but // saying "is a directory" beats saying "does not exist" about a path // the user can see with their own eyes. + // D3 says an ABSOLUTE path, and this is where that stops being a + // description and becomes a check. A relative locator that happens to + // exist resolves against the current working directory — which is + // precisely the ambient, cwd-dependent resolution D3 forbids: the same + // OWEN_RUST_CORE would select different binaries from different + // directories, and "which binary ran" would stop being a property of + // the configuration. Rejected before any existence test, so the + // diagnostic names the real problem rather than reporting on whatever + // the relative path happened to hit. + if (!Path.IsPathFullyQualified(raw)) + { + throw new RustCoreNotResolvedException( + Problem($"is not an absolute path: '{raw}'. Stage 1 resolves the candidate " + + "from this variable alone, so a path relative to the current directory " + + "would select a different binary depending on where Owen was run")); + } + if (Directory.Exists(raw)) { throw new RustCoreNotResolvedException( diff --git a/scripts/own-check.ps1 b/scripts/own-check.ps1 index 2a6b7400..e6669700 100644 --- a/scripts/own-check.ps1 +++ b/scripts/own-check.ps1 @@ -80,6 +80,53 @@ param( Set-StrictMode -Version Latest $ErrorActionPreference = "Stop" +function Invoke-CapturedProcess { + <# + .SYNOPSIS + Run a child and capture its streams as RAW BYTES. + + .DESCRIPTION + `Start-Process -RedirectStandardOutput` is NOT byte-faithful: measured + against the same input, the Python reference wrote 211 bytes and the + redirected file held 210 — a blank line silently dropped. Compare mode + claims the two engines' PUBLIC BYTES are identical, so capturing the + reference through a lossy channel does not merely lose formatting: it can + manufacture a divergence that does not exist, or hide one that does, and + an agreement reached over a corrupted capture is not an agreement at all. + + This drains both pipes as byte streams, concurrently — a serial read + deadlocks once either pipe fills — which is the same thing the `owen` + launcher does in C#. A failure to START is deliberately allowed to + propagate so the caller can map it to D3.1's configuration exit (2). + #> + param( + [Parameter(Mandatory = $true)][string]$FilePath, + [Parameter(Mandatory = $true)][string[]]$ArgumentList, + [Parameter(Mandatory = $true)][string]$StdoutPath, + [Parameter(Mandatory = $true)][string]$StderrPath + ) + $psi = New-Object System.Diagnostics.ProcessStartInfo + $psi.FileName = $FilePath + foreach ($a in $ArgumentList) { $psi.ArgumentList.Add($a) } + $psi.RedirectStandardOutput = $true + $psi.RedirectStandardError = $true + $psi.UseShellExecute = $false + $proc = [System.Diagnostics.Process]::Start($psi) + $outFs = [System.IO.File]::Create($StdoutPath) + $errFs = [System.IO.File]::Create($StderrPath) + try { + $outTask = $proc.StandardOutput.BaseStream.CopyToAsync($outFs) + $errTask = $proc.StandardError.BaseStream.CopyToAsync($errFs) + $proc.WaitForExit() + [System.Threading.Tasks.Task]::WaitAll(@($outTask, $errTask)) + } + finally { + $outFs.Dispose() + $errFs.Dispose() + } + return $proc.ExitCode +} + # Default root = the Own.NET checkout this script lives in (scripts\..). if ([string]::IsNullOrEmpty($Root)) { $Root = Split-Path -Parent $PSScriptRoot @@ -104,6 +151,15 @@ if ($Engine -eq "rust" -or $Engine -eq "compare") { if ([string]::IsNullOrWhiteSpace($rustCore)) { $problem = "is not set (or is empty)" } + # D3 says an ABSOLUTE path, and this is where that stops being a + # description and becomes a check: a relative locator that happens to exist + # resolves against the current directory, so the same OWEN_RUST_CORE would + # select different binaries depending on where own-check was run. + elseif (-not [System.IO.Path]::IsPathFullyQualified($rustCore)) { + $problem = ("is not an absolute path: '$rustCore' (Stage 1 resolves the candidate from " + + "this variable alone, so a path relative to the current directory would " + + "select a different binary depending on where own-check was run)") + } elseif (Test-Path -LiteralPath $rustCore -PathType Container) { $problem = "points at a directory, not a file: '$rustCore'" } @@ -111,9 +167,11 @@ if ($Engine -eq "rust" -or $Engine -eq "compare") { $problem = "points at a path that does not exist: '$rustCore'" } if ($problem -ne "") { - # Windows has no execute bit: an existing regular file is accepted here - # and a genuinely broken image fails at spawn, which is the other side - # of the D3.1 seam and already maps to the internal-error path. + # Windows has no execute bit, so an existing regular file is accepted + # here and a file the loader cannot start is caught at SPAWN instead — + # and that is still the locator's side of D3.1's seam ("cannot select + # the candidate"), so it maps to this same configuration exit 2, not to + # the internal-error path. See Invoke-RustCandidate below. [Console]::Error.WriteLine(("own-check: --engine $Engine needs the candidate ``own-cli`` binary, but " + "OWEN_RUST_CORE $problem. Set OWEN_RUST_CORE to the absolute path of the ``own-cli`` " + "executable to run. Owen did not fall back to Python.")) @@ -153,8 +211,21 @@ try { elseif ($Engine -eq "rust") { # The PRODUCTION Rust executable, never own-shadow-engine. $rustArgs = @("ownir") + $ownirArgs - & $rustCore @rustArgs - $rc = $LASTEXITCODE + try { + & $rustCore @rustArgs + $rc = $LASTEXITCODE + } + catch { + # D3.1's seam: the candidate never STARTED — an existing file the + # loader will not run. That is "cannot select the candidate", so it + # is a configuration error (2), not Owen failing internally (5). + # On Windows this is the only point at which a non-runnable + # candidate can be detected, since there is no execute bit to test. + [Console]::Error.WriteLine(("own-check: the candidate ``own-cli`` binary could not be started: " + + "'$rustCore' ($($_.Exception.Message)). Set OWEN_RUST_CORE to a runnable ``own-cli`` " + + "executable. Owen did not fall back to Python.")) + exit 2 + } # 0/1/2 are verdicts and pass through; anything else is not a verdict # and takes the public internal-error path (5) with the raw child # status named. It never runs Python instead. @@ -212,16 +283,27 @@ try { # PowerShell's own string pipeline. $pyArgs = @("-m", "ownlang", "ownir", $pyIn, "--format", $Format, "--severity", $Severity, "--verbosity", $Verbosity) - $p1 = Start-Process -FilePath "python" -ArgumentList $pyArgs -NoNewWindow -Wait -PassThru ` - -RedirectStandardOutput (Join-Path $cmpDir "python.out") ` - -RedirectStandardError (Join-Path $cmpDir "python.err") + $pyRcCaptured = Invoke-CapturedProcess -FilePath "python" -ArgumentList $pyArgs ` + -StdoutPath (Join-Path $cmpDir "python.out") ` + -StderrPath (Join-Path $cmpDir "python.err") $rsArgs = @("ownir", $rsIn, "--format", $Format, "--severity", $Severity, "--verbosity", $Verbosity) - $p2 = Start-Process -FilePath $rustCore -ArgumentList $rsArgs -NoNewWindow -Wait -PassThru ` - -RedirectStandardOutput (Join-Path $cmpDir "rust.out") ` - -RedirectStandardError (Join-Path $cmpDir "rust.err") - $pyRc = $p1.ExitCode - $rsRc = $p2.ExitCode + try { + $rsRcCaptured = Invoke-CapturedProcess -FilePath $rustCore -ArgumentList $rsArgs ` + -StdoutPath (Join-Path $cmpDir "rust.out") ` + -StderrPath (Join-Path $cmpDir "rust.err") + } + catch { + # Same seam as --engine rust: a candidate that never started is + # a configuration error (2), not a compare execution failure + # (5). The compare did not happen. + [Console]::Error.WriteLine(("own-check: the candidate ``own-cli`` binary could not be started: " + + "'$rustCore' ($($_.Exception.Message)). Set OWEN_RUST_CORE to a runnable ``own-cli`` " + + "executable. Owen did not fall back to Python.")) + exit 2 + } + $pyRc = $pyRcCaptured + $rsRc = $rsRcCaptured # D4.1 (c): execution failure first — two results are comparable # only once both exist. @@ -232,6 +314,11 @@ try { "$pyRc, rust exit $rsRc). No engine's result was substituted for the other's failure. " + "Reproduction — input sha256 $captureSha, candidate $rustCore, artifacts in $cmpDir")) if (-not $rsLegal) { [Console]::Error.WriteLine("own-check: raw Rust child status: $rsRc") } + # The message above names $cmpDir as the reproduction evidence, + # so the directory has to outlive this process. Pointing a + # reader at a path and then deleting it on the way out is worse + # than not naming one at all. + $keep = $true exit 5 } @@ -254,8 +341,31 @@ try { exit 5 } - # Agreement: the externally observed result is the reference's. - Get-Content -LiteralPath (Join-Path $cmpDir "python.out") -Raw -ErrorAction SilentlyContinue | Write-Output + # Agreement: the externally observed result is the reference's — + # its RAW BYTES, both streams. `Get-Content -Raw | Write-Output` + # decodes and re-encodes through PowerShell's pipeline, which is + # not the reference's output but a re-rendering of it, and it + # dropped stderr entirely. The reference result is (exit, stdout + # bytes, stderr bytes); C# and own-check.sh both replay all three, + # and this surface now does too. + # + # Implemented for the contract, not for today's statistics: a + # healthy Windows compare currently diverges on CRLF-vs-LF so this + # branch is rarely reached there, but a format, a runtime version + # or the A/B/C resolution can make it reachable, and a replay that + # is wrong only when it finally runs is worse than no replay. + $outBytes = [System.IO.File]::ReadAllBytes((Join-Path $cmpDir "python.out")) + $errBytes = [System.IO.File]::ReadAllBytes((Join-Path $cmpDir "python.err")) + if ($outBytes.Length -gt 0) { + $stdoutStream = [System.Console]::OpenStandardOutput() + $stdoutStream.Write($outBytes, 0, $outBytes.Length) + $stdoutStream.Flush() + } + if ($errBytes.Length -gt 0) { + $stderrStream = [System.Console]::OpenStandardError() + $stderrStream.Write($errBytes, 0, $errBytes.Length) + $stderrStream.Flush() + } $rc = $pyRc } finally { diff --git a/scripts/own-check.sh b/scripts/own-check.sh index d1564d4c..0cf8e8b2 100755 --- a/scripts/own-check.sh +++ b/scripts/own-check.sh @@ -118,8 +118,30 @@ rust_core="" if [[ "$engine" == "rust" || "$engine" == "compare" ]]; then rust_core="${OWEN_RUST_CORE:-}" problem="" + # D3 says an ABSOLUTE path, and this is where that stops being a description + # and becomes a check: a relative locator that happens to exist resolves + # against the current working directory, so the same OWEN_RUST_CORE would + # select different binaries from different directories. + # + # "Absolute" is not one shape here. This script runs under git-bash on + # Windows as well as a POSIX shell, so a genuinely absolute locator may + # arrive as `/d/a/...` (the MSYS form, which is what CI passes), as + # `C:\...` or `C:/...` (a native Windows path), or as a `//server/share` + # UNC. A bare `/*` test would reject two of those and turn a correct + # configuration into a usage error. This accepts the forms this surface + # actually receives and rejects everything else; it deliberately does NOT + # convert between them — D3 ratified an absolute locator, not a path + # translation policy. + is_absolute=0 + case "$rust_core" in + /*) is_absolute=1 ;; # POSIX, and MSYS's /c/... form + [A-Za-z]:[/\]*) is_absolute=1 ;; # C:\... or C:/... + \\?*) is_absolute=1 ;; # \\server\share (UNC) + esac if [[ -z "$rust_core" ]]; then problem="is not set (or is empty)" + elif [[ "$is_absolute" -eq 0 ]]; then + problem="is not an absolute path: '$rust_core' (Stage 1 resolves the candidate from this variable alone, so a path relative to the current directory would select a different binary depending on where own-check was run)" elif [[ -d "$rust_core" ]]; then problem="points at a directory, not a file: '$rust_core'" elif [[ ! -f "$rust_core" ]]; then diff --git a/scripts/render_checkpoint_status.py b/scripts/render_checkpoint_status.py index 450f3adb..211401e5 100644 --- a/scripts/render_checkpoint_status.py +++ b/scripts/render_checkpoint_status.py @@ -193,6 +193,9 @@ ("Stage 1 — the launcher's `--engine` contract: the default, the candidate " "locator, the Rust child status and the compare result contract", "p022-stage1-1"), + ("Stage 1 — the PowerShell launcher surface, measured on a WINDOWS runner " + "(a PowerShell-targeted mutant is only caught by a Windows catcher)", + "p022-stage1-ps1"), ) SELF = "scripts/render_checkpoint_status.py" diff --git a/tests/helpers/stage1_stub.rs b/tests/helpers/stage1_stub.rs new file mode 100644 index 00000000..c184a16a --- /dev/null +++ b/tests/helpers/stage1_stub.rs @@ -0,0 +1,107 @@ +//! A controllable stand-in for the `own-cli` candidate, for #262 Stage-1 controls. +//! +//! Several Stage-1 controls need a candidate whose exit code and output bytes +//! the test chooses: a compare divergence, a compare execution failure, an +//! agreement (which needs the candidate to emit the reference's exact bytes), +//! and a same-input proof (which needs the candidate to report what it was +//! handed). #261's `fault-injection` build can force a panic or an abort, but +//! it cannot be told to produce a chosen answer, so it cannot express any of +//! those cases. +//! +//! Why a compiled binary rather than a shell script: a `#!`-stub is Unix-only, +//! and the launcher spawns its candidate as a process — on Windows it cannot +//! start a `.cmd`, and git-bash happily runs a mode-644 file, which is how the +//! earlier shell stubs made "non-executable" untestable there. A real native +//! executable behaves the same way on both platforms, which is what lets the +//! compare controls run on Windows at all instead of being declared N/A. +//! +//! Why plain `rustc` and not a cargo crate: adding a workspace member would +//! move the crate-edge DAG that #261's gate pins. This file is compiled +//! directly — +//! +//! ```text +//! rustc -O tests/helpers/stage1_stub.rs -o /stage1-stub[.exe] +//! ``` +//! +//! — so it has no dependencies, no crate, and no effect on the production +//! graph. It is test scaffolding and is never packaged, published, or reachable +//! from any production path. +//! +//! It is invoked exactly like the real candidate (`stub ownir --format +//! F --severity S`) and is configured entirely by environment variables: +//! +//! ```text +//! STAGE1_STUB_EXIT exit with this code (default 0) +//! STAGE1_STUB_STDOUT_FILE write this file's bytes, verbatim, to stdout +//! STAGE1_STUB_STDERR_FILE write this file's bytes, verbatim, to stderr +//! STAGE1_STUB_COPY_INPUT copy the facts argument to this path before exiting +//! STAGE1_STUB_VERSION answer `--version` with this line, exit 0 +//! ``` +//! +//! `STAGE1_STUB_VERSION` lets the stub stand in for the PYTHON side as well as +//! the Rust one. The launcher validates `OWEN_PYTHON` by running it with +//! `--version` and reading "Python X.Y" back, so a stub that always failed +//! would be rejected at resolution (exit 3) and never reach the compare. With +//! this it answers the probe like a supported interpreter and then fails the +//! actual run — which is the only way to exercise "the REFERENCE produced no +//! verdict while the candidate did", and it does so on both platforms rather +//! than through a Unix-only shell wrapper. +//! +//! `STDOUT_FILE`/`STDERR_FILE` take a FILE rather than a string because the +//! agreement control has to reproduce the reference's output byte for byte, +//! including its line endings — the very bytes an environment variable would +//! be least trustworthy about. +//! +//! `COPY_INPUT` copies rather than hashes so this file needs no digest +//! implementation: the harness hashes the copy and compares it against the +//! launcher's attested capture, which is the same measurement with the +//! arithmetic left where a library already exists. + +use std::io::Write as _; + +fn env(name: &str) -> Option { + std::env::var(name).ok().filter(|v| !v.is_empty()) +} + +fn main() { + // argv: ownir --format F --severity S [...] + let args: Vec = std::env::args().skip(1).collect(); + + // The interpreter-probe impersonation, before anything else: the launcher + // asks a candidate interpreter for its version and refuses one it cannot + // read, so this answer has to come before any configured failure. + if let Some(version) = env("STAGE1_STUB_VERSION") { + if args.iter().any(|a| a == "--version") { + println!("{version}"); + std::process::exit(0); + } + } + + if let Some(dest) = env("STAGE1_STUB_COPY_INPUT") { + // args[0] is the subcommand ("ownir"); args[1] is the facts path. + if let Some(facts) = args.get(1) { + let _ = std::fs::copy(facts, dest); + } + } + + if let Some(path) = env("STAGE1_STUB_STDOUT_FILE") { + if let Ok(bytes) = std::fs::read(path) { + let mut out = std::io::stdout().lock(); + let _ = out.write_all(&bytes); + let _ = out.flush(); + } + } + + if let Some(path) = env("STAGE1_STUB_STDERR_FILE") { + if let Ok(bytes) = std::fs::read(path) { + let mut err = std::io::stderr().lock(); + let _ = err.write_all(&bytes); + let _ = err.flush(); + } + } + + let code: i32 = env("STAGE1_STUB_EXIT") + .and_then(|v| v.parse().ok()) + .unwrap_or(0); + std::process::exit(code); +} diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index 5b772f51..c95b7e83 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -233,17 +233,53 @@ def tail(r: subprocess.CompletedProcess[bytes], limit: int = 500) -> str: return " | ".join(parts) or "(both streams empty)" -def write_stub(path: Path, exit_code: int, stdout: str = "", stderr: str = "") -> Path: - """A candidate that exits with a chosen code. Unix only — the real - fault-injection binary covers both platforms for the cases it can force.""" - path.write_text( - "#!/usr/bin/env bash\n" - f"printf '%s' {json.dumps(stdout)}\n" - f"printf '%s' {json.dumps(stderr)} >&2\n" - f"exit {exit_code}\n", - encoding="utf-8") - path.chmod(0o755) - return path +def stub_exe(tmp: Path) -> str | None: + """The controllable native candidate (`tests/helpers/stage1_stub.rs`). + + Built here with plain `rustc` when CI has not already supplied one, which + keeps it out of the cargo workspace and therefore out of #261's crate-edge + DAG gate. It replaces the shebang stubs these controls used to write: those + were Unix-only, so every compare control was declared not-applicable on + Windows — the coverage gap that let three PowerShell defects sit green. A + real executable answers the same on both platforms. + """ + supplied = os.environ.get("OWEN_STAGE1_STUB") + if supplied and Path(supplied).is_file(): + return supplied + if shutil.which("rustc") is None: + return None + out = tmp / ("stage1-stub.exe" if os.name == "nt" else "stage1-stub") + if out.is_file(): + return str(out) + src = ROOT / "tests/helpers/stage1_stub.rs" + r = subprocess.run(["rustc", "-O", str(src), "-o", str(out)], + capture_output=True, check=False) + return str(out) if r.returncode == 0 and out.is_file() else None + + +def stub_env(tmp: Path, name: str, *, exit_code: int | None = None, + stdout: bytes | None = None, stderr: bytes | None = None, + copy_input: Path | None = None, version: str | None = None + ) -> dict[str, str]: + """Configure the native stub for one case. Byte payloads go through FILES, + not environment strings, because the agreement control has to reproduce the + reference's output exactly — line endings included.""" + env: dict[str, str] = {} + if exit_code is not None: + env["STAGE1_STUB_EXIT"] = str(exit_code) + if stdout is not None: + f = tmp / f"{name}.out.bin" + f.write_bytes(stdout) + env["STAGE1_STUB_STDOUT_FILE"] = str(f) + if stderr is not None: + f = tmp / f"{name}.err.bin" + f.write_bytes(stderr) + env["STAGE1_STUB_STDERR_FILE"] = str(f) + if copy_input is not None: + env["STAGE1_STUB_COPY_INPUT"] = str(copy_input) + if version is not None: + env["STAGE1_STUB_VERSION"] = version + return env # --- the controls ---------------------------------------------------------- @@ -333,6 +369,157 @@ def control_bad_locator_is_2(sample: Path, tmp: Path) -> None: ok(check, f"{total} invalid-locator cases all exit 2, no fallback") +def control_absolute_locator_only(sample: Path, tmp: Path) -> None: + """D3: the locator is an ABSOLUTE path, and a relative one is refused. + + A relative locator that happens to exist resolves against the current + working directory, so the same OWEN_RUST_CORE would select a different + binary depending on where Owen ran — the ambient resolution D3 exists to + forbid, and the failure mode ("which binary did we measure?") that D3's no- + discovery rule is about. The candidate here genuinely EXISTS, so nothing + but the absoluteness check can reject it. + + The extractor count is part of the assertion. The locator is a preflight, + so a correct launcher rejects before doing expensive work; a validation + that drifted to after extraction would still exit 2 and still look green + without this. + """ + check = "absolute-locator-only" + core = rust_core() + if core is None or not have_dotnet(): + skip(check, "no OWEN_RUST_CORE/dotnet") + return + + # A real, runnable candidate reachable by a RELATIVE path: copy the + # production binary into a working directory and name it "./". + workdir = tmp / "relative-cwd" + workdir.mkdir(exist_ok=True) + local = workdir / ("own-cli.exe" if os.name == "nt" else "own-cli") + shutil.copy2(core, local) + if os.name != "nt": + local.chmod(0o755) + relative = f".{os.sep}{local.name}" + + tally = tmp / "abs-dotnet.log" + if tally.exists(): + tally.unlink() + shim = tmp / "abs-shim" + shim.mkdir(exist_ok=True) + shim_ok = os.name != "nt" + if shim_ok: + (shim / "dotnet").write_text( + "#!/usr/bin/env bash\n" + f'printf "%s\\n" "$*" >> {json.dumps(str(tally))}\n' + f'exec {json.dumps(str(shutil.which("dotnet")))} "$@"\n', + encoding="utf-8") + (shim / "dotnet").chmod(0o755) + + problems = [] + for engine in ("rust", "compare"): + env = dict(os.environ) + env["OWEN_RUST_CORE"] = relative + if shim_ok: + env["PATH"] = f"{shim}{os.pathsep}{os.environ.get('PATH', '')}" + + r = subprocess.run( + [bash_exe(), str(ROOT / "scripts/own-check.sh"), + "--engine", engine, "--", str(sample)], + capture_output=True, env=env, cwd=str(workdir), check=False) + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode != 2: + problems.append(f"own-check.sh/{engine}: exit {r.returncode}, expected 2 " + f"for a relative locator [{tail(r)}]") + elif "absolute" not in merged: + problems.append(f"own-check.sh/{engine}: rejected without naming the absolute " + "requirement") + if b"OWN001" in r.stdout: + problems.append(f"own-check.sh/{engine}: produced a verdict — the candidate ran, " + "or Python did") + + dll = launcher_dll() + if dll is not None: + r2 = subprocess.run( + ["dotnet", dll, "check", "--engine", engine, str(sample)], + capture_output=True, env=env, cwd=str(workdir), check=False) + merged2 = (r2.stdout + r2.stderr).decode("utf-8", "replace") + if r2.returncode != 2: + problems.append(f"owen/{engine}: exit {r2.returncode}, expected 2 for a " + f"relative locator [{tail(r2)}]") + elif "absolute" not in merged2: + problems.append(f"owen/{engine}: rejected without naming the absolute requirement") + if b"OWN001" in r2.stdout: + problems.append(f"owen/{engine}: produced a verdict for a relative locator") + + if shim_ok: + lines = tally.read_text(encoding="utf-8").splitlines() if tally.exists() else [] + extractions = [ln for ln in lines if "OwnSharp.Extractor" in ln] + if extractions: + problems.append(f"the extractor ran {len(extractions)} time(s) before the locator was " + "rejected — validation drifted past the preflight") + + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "a relative but existing locator is refused with exit 2 on both surfaces, " + "before any extraction") + + +def control_compare_failure_is_classified(sample: Path, tmp: Path) -> None: + """D4.1: the compare verdict is what the case WAS, not what a Rust-child + field happened to be. + + The case the old classifier got wrong: the PYTHON reference produces an + unexpected exit while the candidate answers legally. There is no Rust child + status to record, so a classifier inferring from `child_exit_code` stamped + "divergence" onto evidence whose own diagnostic said "execution failure". + """ + check = "compare-failure-classified" + core = rust_core() + stub = stub_exe(tmp) + if core is None or stub is None or not have_dotnet(): + skip(check, "no OWEN_RUST_CORE / native stub / dotnet") + return + + evidence = Path.home() / ".owen/compare/last-compare.json" + if evidence.exists(): + evidence.unlink() + # The stub stands in for PYTHON: it answers the launcher's version probe + # like a supported interpreter, then fails the actual run. + env = {"OWEN_RUST_CORE": core, + "OWEN_PYTHON": stub, + **stub_env(tmp, "pyfail", exit_code=42, version="Python 3.13.0")} + r = run_owen(["--engine", "compare", "--format", "human", str(sample)], env=env) + if r is None: + skip(check, "no built launcher/dotnet") + return + + problems = [] + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode != 5: + problems.append(f"a Python-only failure exited {r.returncode}, expected public 5 " + f"[{tail(r)}]") + if "execution failure" not in merged: + problems.append("the diagnostic does not call it an execution failure") + if b"OWN001" in r.stdout: + problems.append("a verdict was exposed after the reference failed") + if not evidence.exists(): + problems.append("no compare evidence was written") + else: + try: + data = json.loads(evidence.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + problems.append(f"the compare evidence is unreadable: {exc}") + data = {} + verdict = data.get("verdict") + if verdict != "execution-failure": + problems.append(f"evidence verdict is {verdict!r}, expected 'execution-failure' — " + "the structured record disagrees with the diagnostic beside it") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "a Python-only failure is recorded as execution-failure, not divergence") + + def control_default_stays_python(sample: Path) -> None: """D1: the default engine is Python. Proved NEGATIVELY and positively: a default run with a deliberately unusable Python must fail on Python (the @@ -660,27 +847,18 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None # file it was actually handed, and that digest must equal the capture # digest the launcher attests in its evidence. If compare ever fed the two # engines different bytes, these two values part company. - if os.name == "nt": - not_applicable(same_check, "needs a recording stub candidate with a shebang; Unix-only. " - "The launcher code that materialises and verifies the two " - "engine inputs is platform-independent and is measured on the " - "Linux leg") - return - seen = tmp / "candidate-saw.sha256" - recorder = tmp / "recording-core" - recorder.write_text( - "#!/usr/bin/env bash\n" - "# args: ownir --format F --severity S\n" - f"sha256sum < \"$2\" | cut -d' ' -f1 > {json.dumps(str(seen))}\n" - "exit 0\n", - encoding="utf-8") - recorder.chmod(0o755) + stub = stub_exe(tmp) + if stub is None: + skip(same_check, "no native stub (set OWEN_STAGE1_STUB or provide rustc)") + return + seen = tmp / "candidate-saw.json" + recorder_env = stub_env(tmp, "recorder", exit_code=0, copy_input=seen) evidence = Path.home() / ".owen/compare/last-compare.json" if evidence.exists(): evidence.unlink() r2 = run_owen(["--engine", "compare", "--format", "human", str(sample)], - env={"OWEN_RUST_CORE": str(recorder)}) + env={"OWEN_RUST_CORE": stub, **recorder_env}) if r2 is None: skip(same_check, "no built launcher/dotnet") return @@ -696,7 +874,9 @@ def control_compare_same_input_and_extract_once(sample: Path, tmp: Path) -> None except (OSError, json.JSONDecodeError) as exc: fail(same_check, f"the compare evidence is unreadable: {exc}") return - candidate_saw = seen.read_text(encoding="utf-8").strip() + # The stub copied the bytes it was handed; hashing the copy here is the + # same measurement with the arithmetic left where a library exists. + candidate_saw = hashlib.sha256(seen.read_bytes()).hexdigest() if candidate_saw != attested: fail(same_check, f"the candidate was handed bytes hashing to {candidate_saw}, but the " f"launcher attested the capture as {attested} — the engines did not " @@ -758,12 +938,10 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: div_check = "divergence-is-5" exec_check = "exec-failure-is-5" sub_check = "compare-no-substitution" - if os.name == "nt": + stub = stub_exe(tmp) + if stub is None: for c in (div_check, exec_check, sub_check): - not_applicable(c, "needs a synthetic candidate with chosen output; a shebang stub " - "is Unix-only, and the launcher cannot spawn a .cmd. The C# and " - "bash logic under test is platform-independent and is measured on " - "the Linux leg") + skip(c, "no native stub (set OWEN_STAGE1_STUB or provide rustc)") return if not have_dotnet(): for c in (div_check, exec_check, sub_check): @@ -775,12 +953,13 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: # in the bytes. That is deliberate: a stub that also differed in its exit # code would let a compare that had stopped comparing stdout still look # correct, because the exit-code check alone would flag the divergence. - diverging = write_stub(tmp / "diverging-core", 1, stdout="a different answer\n") + diverging = {"OWEN_RUST_CORE": stub, + **stub_env(tmp, "diverging", exit_code=1, stdout=b"a different answer\n")} div_runs = [("own-check.sh", run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], - env={"OWEN_RUST_CORE": str(diverging)}))] + env=diverging))] owen_div = run_owen(["--engine", "compare", "--format", "human", str(sample)], - env={"OWEN_RUST_CORE": str(diverging)}) + env=diverging) if owen_div is not None: div_runs.append(("owen", owen_div)) else: @@ -820,12 +999,14 @@ def control_compare_failure_and_divergence(sample: Path, tmp: Path) -> None: "evidence, on every surface") # (c) execution failure: a candidate that produces no verdict at all. - crashing = write_stub(tmp / "crashing-core", 42, stderr="forced execution failure\n") + crashing = {"OWEN_RUST_CORE": stub, + **stub_env(tmp, "crashing", exit_code=42, + stderr=b"forced execution failure\n")} exec_runs = [("own-check.sh", run_own_check(["--engine", "compare", "--format", "human", "--", str(sample)], - env={"OWEN_RUST_CORE": str(crashing)}))] + env=crashing))] owen_exec = run_owen(["--engine", "compare", "--format", "human", str(sample)], - env={"OWEN_RUST_CORE": str(crashing)}) + env=crashing) if owen_exec is not None: exec_runs.append(("owen", owen_exec)) else: @@ -911,6 +1092,8 @@ def run() -> int: # No fail-fast: every control runs, so a campaign sees every catcher a # mutation trips rather than only the first. control_bad_locator_is_2(sample_dir, tmp) + control_absolute_locator_only(sample_dir, tmp) + control_compare_failure_is_classified(sample_dir, tmp) control_no_selector_in_own_cli() control_default_stays_python(sample_dir) control_rust_actually_runs_rust(sample_dir) diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py new file mode 100644 index 00000000..f933c79c --- /dev/null +++ b/tests/test_stage1_ps1.py @@ -0,0 +1,349 @@ +#!/usr/bin/env python3 +"""#262 Stage 1 — `scripts/own-check.ps1`'s engine contract, driven. + +Why this exists as its own harness. `tests/test_stage1_engine.py` drives the +`owen` launcher and `own-check.sh`; the PowerShell surface was only ever +touched by a CI smoke step (healthy `-Engine rust`, and a NONEXISTENT +OWEN_RUST_CORE). That is real coverage, but it never reaches compare, +agreement replay, an existing-but-unstartable candidate, or execution-failure +evidence — which is exactly why three PowerShell defects sat green through a +16/16 mutation campaign. Absence of a control is not evidence of correctness, +and a campaign can only prove what some control actually observes. + +The controls here are the PowerShell halves of the ratified rulings: + + ps1-absolute-locator D3 a relative but existing locator is refused + ps1-not-started-is-2 D3.1 an existing file the loader will not start + is a configuration error (2), not 5 + ps1-agreement-replays D4.1a agreement replays the reference's RAW bytes + on BOTH streams, not a re-encoded stdout + ps1-failure-evidence D4.1c the reproduction evidence it names still + exists after the process exits + +Platform. These drive `pwsh`, which runs on Linux too, and every control here +is written to be platform-neutral so it can be developed and debugged +anywhere. That convenience does NOT make a Linux run acceptable as evidence: a +mutation whose target is `scripts/own-check.ps1` is only `caught` when a +WINDOWS PowerShell catcher observes the mutant and fails. The campaign that +owns these mutants therefore runs on a Windows runner +(`.github/workflows/ci.yml`, the `stage1-ps1-mutations` job), and a Linux run +of this file is a developer convenience, never the record. + +Failures print `FAIL[]: `; nothing stops at the first one. + +Run: python tests/test_stage1_ps1.py +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +SAMPLE_CS = """using System; +using System.IO; + +public class Leaky +{ + public void Run() + { + var s = new FileStream("x.txt", FileMode.OpenOrCreate); + Console.WriteLine(s.Length); + } +} +""" + +_FAILURES: list[tuple[str, str]] = [] +_PASSES: list[str] = [] +_SKIPS: list[tuple[str, str]] = [] + + +def fail(check: str, detail: str) -> None: + _FAILURES.append((check, detail)) + print(f"FAIL[{check}]: {detail}") + + +def ok(check: str, detail: str = "") -> None: + _PASSES.append(check) + print(f"ok[{check}]{': ' + detail if detail else ''}") + + +def skip(check: str, why: str) -> None: + """Under OWEN_STAGE1_REQUIRE a skip is a failure: the job that sets it + exists to supply the toolchain, so a control that could not run there is a + denominator that quietly shrank.""" + if os.environ.get("OWEN_STAGE1_REQUIRE") == "1": + fail(check, f"required control could not run: {why}") + return + _SKIPS.append((check, why)) + print(f"skip[{check}]: {why}") + + +def tail(r: subprocess.CompletedProcess[bytes], limit: int = 400) -> str: + err = r.stderr.decode("utf-8", "replace").strip() + out = r.stdout.decode("utf-8", "replace").strip() + parts = [] + if err: + parts.append(f"stderr: …{err[-limit:]}") + if out: + parts.append(f"stdout: …{out[-limit:]}") + return " | ".join(parts) or "(both streams empty)" + + +# --- toolchain ------------------------------------------------------------- + + +def pwsh_exe() -> str | None: + return shutil.which("pwsh") or shutil.which("powershell") + + +def rust_core() -> str | None: + p = os.environ.get("OWEN_RUST_CORE") + return p if p and Path(p).is_file() else None + + +def stub_exe(tmp: Path) -> str | None: + """The controllable native candidate, shared with the other harness.""" + supplied = os.environ.get("OWEN_STAGE1_STUB") + if supplied and Path(supplied).is_file(): + return supplied + if shutil.which("rustc") is None: + return None + out = tmp / ("stage1-stub.exe" if os.name == "nt" else "stage1-stub") + if out.is_file(): + return str(out) + r = subprocess.run( + ["rustc", "-O", str(ROOT / "tests/helpers/stage1_stub.rs"), "-o", str(out)], + capture_output=True, check=False) + return str(out) if r.returncode == 0 and out.is_file() else None + + +def run_ps1(args: list[str], env: dict[str, str] | None = None, + cwd: str | None = None) -> subprocess.CompletedProcess[bytes] | None: + """Drive own-check.ps1, capturing RAW bytes — the replay contract is about + bytes, so the harness must not decode on the way in either.""" + pwsh = pwsh_exe() + if pwsh is None: + return None + e = dict(os.environ) + e.update(env or {}) + return subprocess.run( + [pwsh, "-NoLogo", "-NoProfile", "-File", str(ROOT / "scripts/own-check.ps1"), *args], + capture_output=True, env=e, cwd=cwd or str(ROOT), check=False) + + +# --- controls -------------------------------------------------------------- + + +def control_absolute_locator(sample: Path, tmp: Path) -> None: + """D3: a relative locator resolves against the working directory, so the + same variable would select different binaries from different places.""" + check = "ps1-absolute-locator" + core = rust_core() + if core is None: + skip(check, "no OWEN_RUST_CORE") + return + workdir = tmp / "ps1-relative-cwd" + workdir.mkdir(exist_ok=True) + local = workdir / ("own-cli.exe" if os.name == "nt" else "own-cli") + shutil.copy2(core, local) + if os.name != "nt": + local.chmod(0o755) + + problems = [] + for engine in ("rust", "compare"): + r = run_ps1(["-Engine", engine, "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": f".{os.sep}{local.name}"}, cwd=str(workdir)) + if r is None: + skip(check, "no pwsh") + return + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode != 2: + problems.append(f"{engine}: exit {r.returncode}, expected 2 [{tail(r)}]") + elif "absolute" not in merged: + problems.append(f"{engine}: refused without naming the absolute requirement") + if b"OWN001" in r.stdout: + problems.append(f"{engine}: produced a verdict for a relative locator") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "a relative but existing locator is refused with exit 2") + + +def control_not_started_is_2(sample: Path, tmp: Path) -> None: + """D3.1: a candidate that EXISTS but the loader will not start never got to + run, so it is on the locator's side of the seam — exit 2, not 5. + + This is the case the CI smoke step could not reach: it used a NONEXISTENT + path, which `Test-Path` rejects long before any spawn. + """ + check = "ps1-not-started-is-2" + unstartable = tmp / "not-a-program.txt" + unstartable.write_text("this is text, not an executable image\n", encoding="utf-8") + + problems = [] + for engine in ("rust", "compare"): + r = run_ps1(["-Engine", engine, "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": str(unstartable.resolve())}) + if r is None: + skip(check, "no pwsh") + return + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode == 5: + problems.append(f"{engine}: exit 5 — a candidate that never started was reported as " + "an internal failure instead of a configuration error") + elif r.returncode != 2: + problems.append(f"{engine}: exit {r.returncode}, expected 2 [{tail(r)}]") + elif "could not be started" not in merged: + problems.append(f"{engine}: exit 2 without saying the candidate could not be started") + if b"OWN001" in r.stdout: + problems.append(f"{engine}: produced a verdict — Python answered for the candidate") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "an existing-but-unstartable candidate is exit 2 on both engine paths") + + +def control_agreement_replays_bytes(sample: Path, tmp: Path) -> None: + """D4.1(a): on agreement the external result is the REFERENCE's — its raw + bytes, on both streams. + + Agreement is manufactured deliberately: the candidate is handed the exact + bytes the Python reference produces for this input, so the two engines + genuinely agree. That is the only way to reach this branch on Windows, + where a real candidate diverges from the reference on CRLF alone — and the + branch has to be right for when it becomes reachable, not merely for as + long as it is rare. + """ + check = "ps1-agreement-replays" + stub = stub_exe(tmp) + if stub is None: + skip(check, "no native stub (set OWEN_STAGE1_STUB or provide rustc)") + return + if shutil.which("dotnet") is None or shutil.which("python") is None: + skip(check, "no dotnet/python") + return + + # Extract once, then ask the reference what it says about those facts. + facts = tmp / "agree.facts.json" + ex = subprocess.run( + [str(ROOT / "scripts/own-check.sh"), "--emit-facts", str(facts), "--", str(sample)], + capture_output=True, check=False, cwd=str(ROOT), + env={**os.environ, "PYTHONPATH": str(ROOT)}) + if not facts.is_file(): + skip(check, f"could not extract facts to drive the reference [{tail(ex)}]") + return + ref = subprocess.run( + ["python", "-m", "ownlang", "ownir", str(facts), + "--format", "human", "--severity", "error", "--verbosity", "normal"], + capture_output=True, check=False, cwd=str(ROOT), + env={**os.environ, "PYTHONPATH": str(ROOT)}) + + env = {"OWEN_RUST_CORE": stub, + "STAGE1_STUB_EXIT": str(ref.returncode)} + out_file = tmp / "ref.out.bin" + err_file = tmp / "ref.err.bin" + out_file.write_bytes(ref.stdout) + err_file.write_bytes(ref.stderr) + env["STAGE1_STUB_STDOUT_FILE"] = str(out_file) + env["STAGE1_STUB_STDERR_FILE"] = str(err_file) + + r = run_ps1(["-Engine", "compare", "-Format", "human", str(sample)], env=env) + if r is None: + skip(check, "no pwsh") + return + if r.returncode not in (0, 1): + skip(check, f"the engines did not agree, so the replay branch was not reached " + f"(exit {r.returncode}) [{tail(r)}]") + return + + problems = [] + if r.stdout != ref.stdout: + problems.append(f"stdout replay is not byte-faithful: {len(r.stdout)} bytes replayed vs " + f"{len(ref.stdout)} from the reference") + if ref.stderr and r.stderr != ref.stderr: + problems.append(f"stderr replay is not byte-faithful: {len(r.stderr)} bytes replayed vs " + f"{len(ref.stderr)} from the reference (a dropped stderr reads as a " + "silent run)") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, f"agreement replays the reference's raw bytes on both streams " + f"({len(ref.stdout)} out, {len(ref.stderr)} err)") + + +def control_failure_evidence_exists(sample: Path, tmp: Path) -> None: + """D4.1(c): the reproduction evidence a failure NAMES has to survive it. + + Pointing a reader at a directory and deleting it on the way out is worse + than naming nothing: the message reads as reproducible and is not. + """ + check = "ps1-failure-evidence" + stub = stub_exe(tmp) + if stub is None: + skip(check, "no native stub (set OWEN_STAGE1_STUB or provide rustc)") + return + r = run_ps1(["-Engine", "compare", "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": stub, "STAGE1_STUB_EXIT": "42"}) + if r is None: + skip(check, "no pwsh") + return + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + if r.returncode != 5: + fail(check, f"a compare execution failure exited {r.returncode}, expected 5 [{tail(r)}]") + return + + problems = [] + if "42" not in merged: + problems.append("the raw candidate status is not retained in the diagnostic") + # Whatever the message advertises as evidence must still be there. A path + # is only acceptable if it survives; otherwise the message must carry the + # reproduction inline. + named = [tok.strip().rstrip(".,") for tok in merged.replace("\n", " ").split() + if ("owen-compare-" in tok)] + missing = [n for n in named if not Path(n).exists()] + if missing: + problems.append(f"names evidence that no longer exists after exit: {missing[:2]}") + if not named and "sha256" not in merged: + problems.append("names neither a surviving artifact directory nor an inline " + "reproduction (input digest)") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, "the advertised reproduction evidence survives the process") + + +# --- harness --------------------------------------------------------------- + + +def run() -> int: + if pwsh_exe() is None: + skip("ps1-harness", "no pwsh on this machine") + print("\nstage-1 ps1 controls: no PowerShell available") + return 1 if _FAILURES else 0 + + with tempfile.TemporaryDirectory(prefix="owen-stage1-ps1-") as td: + tmp = Path(td) + sample_dir = tmp / "sample" + sample_dir.mkdir() + (sample_dir / "Leak.cs").write_text(SAMPLE_CS, encoding="utf-8") + + control_absolute_locator(sample_dir, tmp) + control_not_started_is_2(sample_dir, tmp) + control_agreement_replays_bytes(sample_dir, tmp) + control_failure_evidence_exists(sample_dir, tmp) + + print() + print(f"stage-1 ps1 controls: {len(_PASSES)} passed, {len(_FAILURES)} failed, " + f"{len(_SKIPS)} skipped") + for name, why in _SKIPS: + print(f" skip {name}: {why}") + return 1 if _FAILURES else 0 + + +if __name__ == "__main__": + sys.exit(run()) From 8a68b80a96c8e44674220432e20666cb494eea23 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:12:27 +0000 Subject: [PATCH 15/22] test(stage1): the ps1 spawn seam is a question only Windows can be asked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Linux leg of `tests (py3.13)` failed on ps1-not-started-is-2, and the reason is worth recording rather than routing around. PowerShell on Linux does not refuse a non-executable file: it hands it to the DESKTOP OPENER. The run exits 0 with `xdg-open: no method available for opening ...`, so the platform answered a different question — "can something display this?" — and answered it successfully. "The loader will not start this image" is not a state Linux can be put in. The part worth admitting: this control PASSED on my development container, for the wrong reason. No xdg-open is installed there, so the invocation failed and looked exactly like a refusal. A control whose verdict turns on whether a desktop helper happens to be installed is not measuring the contract, and I would have shipped it believing it green. That is precisely the case the review's Windows-native rule exists for, now demonstrated rather than asserted: a PowerShell-targeted mutant evaluated on Linux can be "killed" by an accident of the host. The control is therefore declared not-applicable off Windows, with the measured reason, and stays required on Windows where the ps1 mutation campaign runs. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- tests/test_stage1_ps1.py | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py index f933c79c..869abd3c 100644 --- a/tests/test_stage1_ps1.py +++ b/tests/test_stage1_ps1.py @@ -60,6 +60,7 @@ _FAILURES: list[tuple[str, str]] = [] _PASSES: list[str] = [] _SKIPS: list[tuple[str, str]] = [] +_NOT_APPLICABLE: list[tuple[str, str]] = [] def fail(check: str, detail: str) -> None: @@ -83,6 +84,13 @@ def skip(check: str, why: str) -> None: print(f"skip[{check}]: {why}") +def not_applicable(check: str, why: str) -> None: + """A control this platform cannot be asked, as distinct from one whose + toolchain is missing. Printed and counted, never silently dropped.""" + _NOT_APPLICABLE.append((check, why)) + print(f"n/a[{check}]: {why}") + + def tail(r: subprocess.CompletedProcess[bytes], limit: int = 400) -> str: err = r.stderr.decode("utf-8", "replace").strip() out = r.stdout.decode("utf-8", "replace").strip() @@ -182,6 +190,25 @@ def control_not_started_is_2(sample: Path, tmp: Path) -> None: path, which `Test-Path` rejects long before any spawn. """ check = "ps1-not-started-is-2" + if os.name != "nt": + # Measured on a Linux runner: PowerShell there does not refuse a + # non-executable file, it hands it to the DESKTOP OPENER — the run + # exits 0 with `xdg-open: no method available for opening ...`. So + # "the loader will not start this image" is not a state Linux can be + # asked about; it answers a different question and answers it + # successfully. + # + # Worth recording why this was nearly missed: the control PASSED on a + # developer container, for the wrong reason — no xdg-open was + # installed there, so the invocation failed and looked like a refusal. + # A control whose verdict depends on whether a desktop helper happens + # to be present is not measuring the contract. This is exactly the + # case the Windows-native mutation leg exists for. + not_applicable(check, "PowerShell on Linux routes a non-executable file to the desktop " + "opener instead of refusing it, so the spawn seam cannot be posed " + "here; it is required on Windows, where the ps1 mutation campaign " + "runs") + return unstartable = tmp / "not-a-program.txt" unstartable.write_text("this is text, not an executable image\n", encoding="utf-8") @@ -339,9 +366,11 @@ def run() -> int: print() print(f"stage-1 ps1 controls: {len(_PASSES)} passed, {len(_FAILURES)} failed, " - f"{len(_SKIPS)} skipped") + f"{len(_SKIPS)} skipped, {len(_NOT_APPLICABLE)} not applicable on this platform") for name, why in _SKIPS: print(f" skip {name}: {why}") + for name, why in _NOT_APPLICABLE: + print(f" n/a {name}: {why}") return 1 if _FAILURES else 0 From a565c00b72fea94294e4a9ab56daf1d44eec9778 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:27:42 +0000 Subject: [PATCH 16/22] fix(stage1): the shell's absoluteness test matched nothing, and only Windows could say so MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows CI rejected every correct configuration with "OWEN_RUST_CORE is not an absolute path", naming the path it had just been handed. The cause is one character in the fix I shipped for review defect 1: [A-Za-z]:[/\]*) Inside a bracket expression the shell's pattern matcher treats `\` as an escape, so `\]` escapes the closing bracket, the set is never terminated, and the arm matches NEITHER `C:/` nor `C:\`. The drive-rooted branch was dead on arrival. Six controls failed on the Windows leg; all six are the same line. Why no test saw it. Every assertion asked one direction — "is a RELATIVE locator refused?" — and a validator that refuses everything answers that perfectly. The positive direction was never asserted, so on Linux, where the locator is `/home/...` and the POSIX arm carried it, nothing was wrong to see. Two things follow, and both are production changes, not test changes. The bracket is fixed and the shapes are gated on the platform. `C:/rust/own-cli` is absolute where Windows resolves it and is a directory called `C:` in the current directory anywhere else, so accepting it everywhere would have left defect 1 half-fixed on this surface: the Linux reading is exactly the ambient resolution D3 forbids. The accepted set now matches what .NET's IsPathFullyQualified accepts on the same host, which is what the other two implementations call, so all three agree on every shape. Drive-relative `C:own-cli.exe` and root-relative `\own-cli.exe` are refused by all three: each still resolves against ambient state. The control asserts both directions. `locator-shapes` drives eight shapes through every implementation available on the host and asserts the REASON rather than the exit code — all eight exit 2 either way, and "refused as not absolute" versus "absolute, but nothing is there" is the whole question. It also fails when two implementations disagree about a shape, which is the same "which binary did we measure?" ambiguity under another name. Every path is absent, so it needs no candidate, no .NET and no spawn. Mutations, one per independent implementation, for the direction that was missing: M20 (own-check.sh POSIX arm), M21 (RustCoreLocator.cs inverted), P06 (own-check.ps1 inverted). And P05, which restores the exact typo above. P05 is the reason the ps1 campaign is now the WINDOWS campaign. A drive-rooted path is absolute only where Windows resolves it, so a mutant of that arm changes nothing a Linux runner can observe — the review's rule about PowerShell-targeted mutants is not about PowerShell, it is about evaluating a mutant where its contract lives, and one arm of a shell script turns out to live there too. The campaign gains a second layer that narrows the shared harness to `locator-shapes`; it needs no candidate and no .NET build, so the Windows leg stays a shell-and-PowerShell job. `OWEN_STAGE1_ONLY` narrows the harness and can never widen it, and an unknown name is an error rather than an empty run: a filter that silently selects nothing is a way to report a green campaign that measured nothing. Campaign layers can now carry `env` in the definition, so a narrowed layer says so where the evidence is read instead of in a wrapper script. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .github/workflows/ci.yml | 27 ++-- docs/evidence/p022-stage1-1.json | 22 +++ ...age1-ps1.json => p022-stage1-windows.json} | 40 +++++- docs/generated/p022-stage1-mutations.md | 24 +++- scripts/mutate_campaign.py | 14 +- scripts/own-check.sh | 45 +++++-- scripts/render_checkpoint_status.py | 8 +- tests/test_stage1_engine.py | 125 ++++++++++++++++-- tests/test_stage1_ps1.py | 36 ++++- 9 files changed, 295 insertions(+), 46 deletions(-) rename docs/evidence/{p022-stage1-ps1.json => p022-stage1-windows.json} (51%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 64656f4b..5066aa2f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -186,7 +186,7 @@ jobs: # The PowerShell surface's own controls. They run on BOTH legs — the # logic is platform-neutral and a Linux run catches regressions early — # but only the Windows leg is evidence for a PowerShell-targeted - # mutation, which is what the stage1-ps1-mutations job below settles. + # mutation, which is what the stage1-windows-mutations job below settles. - name: Stage-1 PowerShell controls env: OWEN_STAGE1_REQUIRE: "1" @@ -256,11 +256,16 @@ jobs: # cannot be settled there at all: a mutant that runs where its control is # weakest is decorative, and proves nothing about the surface it edits. # + # The same is true of one arm of own-check.sh. A drive-rooted path is + # absolute only where Windows resolves it, so a mutant of that arm changes + # nothing a Linux runner can observe, and the Linux campaign would record it + # as caught or survived on the strength of a question it never asked. + # # So this campaign runs here, on Windows, and this job is the gate. It fails # unless every mutation is caught with its expected catcher and the # honesty control survives the unmutated tree. - stage1-ps1-mutations: - name: own-check.ps1 mutation campaign (Windows-native) + stage1-windows-mutations: + name: Windows-native mutation campaign (own-check.ps1 + the drive-rooted locator arm) runs-on: windows-latest defaults: run: @@ -281,26 +286,26 @@ jobs: run: cargo build -p own-cli --release - name: Build the Stage-1 stub candidate run: rustc -O tests/helpers/stage1_stub.rs -o "$RUNNER_TEMP/stage1-stub.exe" - - name: Run the PowerShell mutation campaign + - name: Run the Windows-native mutation campaign env: OWEN_STAGE1_REQUIRE: "1" run: | export OWEN_RUST_CORE="$PWD/rust/target/release/own-cli.exe" export OWEN_STAGE1_STUB="$RUNNER_TEMP/stage1-stub.exe" - python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-ps1.json --run + python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run # The recorded run, printed in full so its provenance and per-mutation # catchers can be read off this job rather than taken on trust — and so # the committed record can be reproduced from a named CI run, the way # #260's sweep record is. - name: Print the recorded result if: always() - run: cat docs/evidence/p022-stage1-ps1.result.json + run: cat docs/evidence/p022-stage1-windows.result.json - name: Assert every mutation was caught by a Windows catcher run: | python - <<'PY' import json, sys - d = json.load(open("docs/evidence/p022-stage1-ps1.result.json", encoding="utf-8")) - defn = json.load(open("docs/evidence/p022-stage1-ps1.json", encoding="utf-8")) + d = json.load(open("docs/evidence/p022-stage1-windows.result.json", encoding="utf-8")) + defn = json.load(open("docs/evidence/p022-stage1-windows.json", encoding="utf-8")) exp = {m["id"]: set(m["expected_catchers"]) for m in defn["mutations"]} problems = [] if d["control"]["outcome"] != "survived": @@ -310,9 +315,9 @@ jobs: problems.append(f"{m['id']}: {m['outcome']}") elif not exp[m["id"]] <= set(m["catchers"]): problems.append(f"{m['id']}: expected catchers missed ({m['catchers']})") - elif not any(c.startswith("ps1::") for c in m["catchers"]): - problems.append(f"{m['id']}: no PowerShell catcher observed it") - print("\n".join(problems) if problems else "every ps1 mutation caught by a Windows catcher") + print("\n".join(problems) if problems + else f"all {len(d['mutations'])} mutations caught natively on Windows, " + "each by the catcher its definition names") sys.exit(1 if problems else 0) PY diff --git a/docs/evidence/p022-stage1-1.json b/docs/evidence/p022-stage1-1.json index 2a238fd6..26e3b21a 100644 --- a/docs/evidence/p022-stage1-1.json +++ b/docs/evidence/p022-stage1-1.json @@ -231,6 +231,28 @@ "expected_catchers": [ "stage1::compare-failure-classified" ] + }, + { + "id": "M20", + "rule": "absolute-locator-is-accepted", + "description": "the shell's POSIX arm stops recognising an absolute path — the over-rejection direction of D3, where a validator that refuses EVERYTHING passes every 'reject the relative one' assertion and makes the tool unusable with a correct configuration", + "target": "scripts/own-check.sh", + "pattern": " /\\*\\) is_absolute=1 ;; # POSIX, MSYS's /c/\\.\\.\\. form, and //srv/share", + "replacement": " /nope-not-a-real-prefix*) is_absolute=1 ;;", + "expected_catchers": [ + "stage1::locator-shapes" + ] + }, + { + "id": "M21", + "rule": "absolute-locator-is-accepted", + "description": "the launcher's absoluteness test is inverted — the same over-rejection direction in the C# implementation: every fully qualified locator is refused as 'not absolute' and every relative one is admitted", + "target": "frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs", + "pattern": " if \\(!Path\\.IsPathFullyQualified\\(raw\\)\\)\n \\{", + "replacement": " if (Path.IsPathFullyQualified(raw))\n {", + "expected_catchers": [ + "stage1::locator-shapes" + ] } ] } diff --git a/docs/evidence/p022-stage1-ps1.json b/docs/evidence/p022-stage1-windows.json similarity index 51% rename from docs/evidence/p022-stage1-ps1.json rename to docs/evidence/p022-stage1-windows.json index 8060e9c5..a1eabeeb 100644 --- a/docs/evidence/p022-stage1-ps1.json +++ b/docs/evidence/p022-stage1-windows.json @@ -1,8 +1,8 @@ { "schema": 1, "comment": "GENERATED-BY-HAND definition; the RESULT beside it is recorded by scripts/mutate_campaign.py --run on a WINDOWS runner and the counts are derived from it, never typed.", - "campaign": "p022-stage1-ps1", - "description": "#262 Stage 1 — the PowerShell launcher surface. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Every mutation here targets own-check.ps1 itself, and every one is a plausible MISREADING rather than a syntactic accident — Test-Path answering a question D3 did not ask, a failed spawn read as the engine blowing up, `Get-Content | Write-Output` read as an echo, cleanup read as tidiness.", + "campaign": "p022-stage1-windows", + "description": "#262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. Every mutation here is a plausible MISREADING rather than a syntactic accident.", "layers": [ { "id": "ps1", @@ -12,9 +12,21 @@ "python", "tests/test_stage1_ps1.py" ] + }, + { + "id": "shapes", + "cwd": ".", + "parser": "python-fail", + "command": [ + "python", + "tests/test_stage1_engine.py" + ], + "env": { + "OWEN_STAGE1_ONLY": "locator-shapes" + } } ], - "layers_comment": "WINDOWS-NATIVE BY CONTRACT. A mutation whose target is scripts/own-check.ps1 is only `caught` when a Windows PowerShell catcher observes the mutant and fails; this campaign therefore runs on a windows-latest runner (.github/workflows/ci.yml, job `stage1-ps1-mutations`). Running it on Linux would execute the mutated PowerShell under a different runtime and could not settle the Windows-specific halves — the spawn seam above all — so a Linux result is not evidence for these mutants and is never recorded as one.", + "layers_comment": "WINDOWS-NATIVE BY CONTRACT. A mutation targeting scripts/own-check.ps1, or the Windows arm of scripts/own-check.sh's locator classifier, is only `caught` when a Windows catcher observes the mutant and fails; this campaign therefore runs on a windows-latest runner (.github/workflows/ci.yml, job `stage1-windows-mutations`). Running it on Linux would execute the mutated PowerShell under a different runtime and would leave the drive-rooted arm unreachable, so a Linux result is not evidence for these mutants and is never recorded as one. The `shapes` layer narrows tests/test_stage1_engine.py to its locator classification control: that control needs no candidate and no .NET, so the Windows leg stays a PowerShell-and-shell job rather than a second full build.", "control": { "id": "M00", "description": "harness-honesty control: no mutation at all, which must report zero failing layers" @@ -63,6 +75,28 @@ "expected_catchers": [ "ps1::ps1-failure-evidence" ] + }, + { + "id": "P05", + "rule": "absolute-locator-is-accepted", + "description": "the shell's drive-rooted arm stops matching — this is the defect that actually shipped: `[/\\]` escapes the closing bracket, so the set is unterminated and matches NEITHER `C:/` nor `C:\\`, and every correct Windows locator was refused as 'not absolute' while every Linux control stayed green", + "target": "scripts/own-check.sh", + "pattern": " \\[A-Za-z\\]:\\[/\\\\\\\\\\]\\*\\) is_absolute=1 ;;", + "replacement": " [A-Za-z]:[/\\]*) is_absolute=1 ;;", + "expected_catchers": [ + "shapes::locator-shapes" + ] + }, + { + "id": "P06", + "rule": "absolute-locator-is-accepted", + "description": "own-check.ps1's absoluteness test is inverted — the over-rejection direction on this surface: every fully qualified locator is refused as 'not absolute' and every relative one is admitted, which no assertion that only feeds it a relative path can see", + "target": "scripts/own-check.ps1", + "pattern": " elseif \\(-not \\[System\\.IO\\.Path\\]::IsPathFullyQualified\\(\\$rustCore\\)\\) \\{", + "replacement": " elseif ([System.IO.Path]::IsPathFullyQualified($rustCore)) {", + "expected_catchers": [ + "ps1::ps1-absolute-locator" + ] } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index a7cee27b..ebc6a168 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -1,4 +1,4 @@ - + # P-022 step 8 (#262) Stage 1 — mutation campaigns @@ -8,13 +8,13 @@ Stage 1 makes the Rust core SELECTABLE by the launcher while Python stays the de Campaign `p022-stage1-1` — #262 Stage 1 — the launcher's engine-selection contract: that Python stays the default, that an explicitly selected Rust core actually runs (and needs no Python), that a Rust failure is never a Python success, that an unexpected child status becomes public exit 5 with the raw status retained, that an unusable OWEN_RUST_CORE is a configuration error rather than a fallback, and that compare extracts once, feeds both engines the same bytes, and refuses to answer when they disagree or when either fails. Every mutation is a plausible MISREADING of that contract rather than a syntactic accident: each one would pass a reviewer who had read the stage's summary instead of its rulings. Every mutation edits a PRODUCTION launcher surface, and every declared layer runs for every mutation (no fail-fast). -Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-1.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. +Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-1.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. | measure | value | |--------------------------------------------------|---| | recorded at commit | `0f51868df54041b8a8ed2af1c8b0ce41838009c8` | | layers run (every one, for every mutation) | `stage1` | -| mutations | 16 | +| mutations | 21 | | caught | 16 | | survived | 0 | | compile-error (no evidence either way) | 0 | @@ -23,6 +23,11 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 | caught without every expected catcher | none | | honesty control `M00` (unmutated tree must pass) | survived — as required | +**This run is not evidence:** + +- the recorded result was taken over a different campaign definition (sha256 or campaign name differs) — re-run the campaign +- result/definition mutation sets differ (missing ['M17', 'M18', 'M19', 'M20', 'M21'], unknown []) + | id | rule | mutation | outcome | caught by | |---|---|---|---|---| | M01 | default-is-python | the Stage-1 default engine is Rust — the cutover read as already decided, instead of Python remaining default until Gate G3 | caught | `stage1::default-stays-python` | @@ -41,3 +46,16 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `b27b905145eec08d…`, 16 | M14 | shell-compare-checks-stdout | the shell compare stops comparing stdout — the exit code read as the whole of 'the public result', dropping the bytes the user actually sees | caught | `stage1::compare-no-substitution`
`stage1::divergence-is-5` | | M15 | shell-divergence-is-5 | the shell reports a divergence as exit 1 — the 'something is wrong' tier reached for, when in public Owen 1 already means findings | caught | `stage1::divergence-is-5` | | M16 | shell-zero-document-fails | the shell's zero-document guard is dropped — an empty document read as a legitimately clean agreement | caught | `stage1::compare-zero-document` | +| M17 | locator-must-be-absolute | the launcher accepts a relative OWEN_RUST_CORE — `GetFullPath` reads as 'it resolves the path for me', which it does: against whatever directory Owen happened to run in | **not recorded** | — | +| M18 | shell-locator-must-be-absolute | the shell accepts a relative OWEN_RUST_CORE — the -f/-x tests look like they answer 'is this a usable binary', and they do, for whatever the current directory made of the path | **not recorded** | — | +| M19 | compare-verdict-is-stated | the compare verdict is inferred from the Rust-child field again — `child_exit_code is null` reads as 'no engine crashed', but it is only ever about the RUST child, so a Python-only failure is stamped 'divergence' | **not recorded** | — | +| M20 | absolute-locator-is-accepted | the shell's POSIX arm stops recognising an absolute path — the over-rejection direction of D3, where a validator that refuses EVERYTHING passes every 'reject the relative one' assertion and makes the tool unusable with a correct configuration | **not recorded** | — | +| M21 | absolute-locator-is-accepted | the launcher's absoluteness test is inverted — the same over-rejection direction in the C# implementation: every fully qualified locator is refused as 'not absolute' and every relative one is admitted | **not recorded** | — | + +## Stage 1 — the surfaces only Windows can be asked about: `own-check.ps1`, and the drive-rooted arm of the shell's locator classifier. Measured on a WINDOWS runner, because a mutant of either is invisible to a Linux catcher + +Campaign `p022-stage1-windows` — #262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. Every mutation here is a plausible MISREADING rather than a syntactic accident. + +Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `eeca9e0d6910c849…`, 6 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. + +**No recorded run** is committed (expected at `docs/evidence/p022-stage1-windows.result.json`): the campaign has a definition but no evidence. Nothing below is a number. diff --git a/scripts/mutate_campaign.py b/scripts/mutate_campaign.py index ad772a0a..75e654b6 100644 --- a/scripts/mutate_campaign.py +++ b/scripts/mutate_campaign.py @@ -125,12 +125,18 @@ class Layer: catcher is named by the CHECK it violated rather than by the case that happened to trip first. A non-zero exit with no such line is still a catch, recorded under a name that says so. + + `env` adds variables to the layer's environment. It is part of the + DEFINITION, and therefore part of the recorded evidence, so a layer that + runs a narrowed subset of a harness says so where the campaign is read + rather than in a wrapper script the reader has to go find. """ id: str cwd: str command: tuple[str, ...] parser: str + env: tuple[tuple[str, str], ...] = () PARSERS = ("cargo", "python-fail") @@ -279,8 +285,13 @@ def _layer(obj: object, where: str) -> Layer: parser = _str(obj, "parser", where) if parser not in PARSERS: raise CampaignError(f"{where}: unknown parser {parser!r} (one of {list(PARSERS)})") + raw_env = obj.get("env", {}) + if not (isinstance(raw_env, dict) + and all(isinstance(k, str) and k and isinstance(v, str) for k, v in raw_env.items())): + raise CampaignError(f"{where}: 'env' must be an object of string names to string values") return Layer(id=_str(obj, "id", where), cwd=str(obj.get("cwd", ".")), - command=tuple(str(c) for c in command), parser=parser) + command=tuple(str(c) for c in command), parser=parser, + env=tuple(sorted((str(k), str(v)) for k, v in raw_env.items()))) def _target(data: dict[str, object], path: str) -> tuple[str | None, tuple[Layer, ...]]: @@ -541,6 +552,7 @@ def _run_layer(layer: Layer) -> tuple[list[str], bool, list[str]]: # p022-shadow-acc-2. cargo and this repository's harnesses both emit UTF-8, # and `errors="replace"` means a stray byte costs one character rather than # the whole run. + env.update(dict(layer.env)) r = subprocess.run(list(layer.command), cwd=os.path.join(ROOT, layer.cwd), env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding="utf-8", errors="replace") diff --git a/scripts/own-check.sh b/scripts/own-check.sh index 0cf8e8b2..f8d739c9 100755 --- a/scripts/own-check.sh +++ b/scripts/own-check.sh @@ -125,18 +125,43 @@ if [[ "$engine" == "rust" || "$engine" == "compare" ]]; then # # "Absolute" is not one shape here. This script runs under git-bash on # Windows as well as a POSIX shell, so a genuinely absolute locator may - # arrive as `/d/a/...` (the MSYS form, which is what CI passes), as - # `C:\...` or `C:/...` (a native Windows path), or as a `//server/share` - # UNC. A bare `/*` test would reject two of those and turn a correct - # configuration into a usage error. This accepts the forms this surface - # actually receives and rejects everything else; it deliberately does NOT - # convert between them — D3 ratified an absolute locator, not a path - # translation policy. + # arrive as `/d/a/...` (the MSYS form), as `C:/...` or `C:\...` (a native + # Windows path, which is what a Windows caller and MSYS's own environment + # translation both hand over), or as a UNC path. A bare `/*` test would + # reject the Windows forms and turn a correct configuration into a usage + # error. This accepts the forms this surface actually receives and rejects + # everything else; it deliberately does NOT convert between them — D3 + # ratified an absolute locator, not a path translation policy. + # + # The set of accepted shapes is the same one .NET's IsPathFullyQualified + # accepts, which is what the other two implementations call. In particular a + # DRIVE-RELATIVE `C:own-cli.exe` and a ROOT-RELATIVE `\own-cli.exe` are both + # rejected: each still resolves against ambient state (the drive's current + # directory, the current drive), which is the thing D3 forbids. + # + # The backslash inside the bracket expression is doubled because the shell's + # pattern matcher treats `\` there as an escape: `[/\]` escapes the closing + # bracket, leaving an unterminated set that matches NEITHER `C:/` nor `C:\`. + # That typo shipped once and was caught by Windows CI rejecting every + # correct Windows locator, so it is spelled out rather than left to be + # rediscovered. is_absolute=0 case "$rust_core" in - /*) is_absolute=1 ;; # POSIX, and MSYS's /c/... form - [A-Za-z]:[/\]*) is_absolute=1 ;; # C:\... or C:/... - \\?*) is_absolute=1 ;; # \\server\share (UNC) + /*) is_absolute=1 ;; # POSIX, MSYS's /c/... form, and //srv/share + esac + # The drive and UNC forms are absolute only where Windows is doing the + # resolving. On Linux `C:/rust/own-cli` names a directory called `C:` in the + # current directory — the exact ambient-resolution case D3 forbids — so + # accepting it everywhere would have left the defect half-fixed on this + # surface and disagreed with the two implementations that call + # IsPathFullyQualified. + case "$(uname -s 2>/dev/null)" in + MINGW*|MSYS*|CYGWIN*|Windows_NT) + case "$rust_core" in + [A-Za-z]:[/\\]*) is_absolute=1 ;; # C:/... or C:\... — drive-ROOTED + \\\\*) is_absolute=1 ;; # \\server\share (UNC) + esac + ;; esac if [[ -z "$rust_core" ]]; then problem="is not set (or is empty)" diff --git a/scripts/render_checkpoint_status.py b/scripts/render_checkpoint_status.py index 211401e5..54ba8ae1 100644 --- a/scripts/render_checkpoint_status.py +++ b/scripts/render_checkpoint_status.py @@ -193,9 +193,11 @@ ("Stage 1 — the launcher's `--engine` contract: the default, the candidate " "locator, the Rust child status and the compare result contract", "p022-stage1-1"), - ("Stage 1 — the PowerShell launcher surface, measured on a WINDOWS runner " - "(a PowerShell-targeted mutant is only caught by a Windows catcher)", - "p022-stage1-ps1"), + ("Stage 1 — the surfaces only Windows can be asked about: `own-check.ps1`, " + "and the drive-rooted arm of the shell's locator classifier. Measured on " + "a WINDOWS runner, because a mutant of either is invisible to a Linux " + "catcher", + "p022-stage1-windows"), ) SELF = "scripts/render_checkpoint_status.py" diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index c95b7e83..67a0a985 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -464,6 +464,77 @@ def control_absolute_locator_only(sample: Path, tmp: Path) -> None: "before any extraction") +def control_locator_shapes(sample: Path) -> None: + """D3, the direction a rejection test cannot see: an ABSOLUTE locator must + be ACCEPTED. + + A validator that refuses everything passes "reject the relative one" + perfectly. That is not a hypothetical: own-check.sh shipped `[/\\]` as its + drive-rooted arm, where the backslash escapes the closing bracket and + leaves an unterminated set matching NEITHER `C:/` nor `C:\\`. Every correct + Windows configuration was refused as "not absolute", and every Linux + control stayed green, because only on Windows is the locator drive-rooted. + + Every path below is ABSENT, so each run stops at the same preflight and the + assertion is on the REASON rather than the exit code — all of these exit 2 + either way. That keeps the control fast, needs no candidate and no .NET, + and makes "refused as not absolute" and "absolute, but nothing is there" + distinguishable, which is the whole question. The UNC row uses the `\\\\.\\` + device form: UNC-shaped to both classifiers, but resolved locally, so it + cannot stall on a name lookup for a server a test invented. + + Which shapes are absolute is a property of the PLATFORM, not of Owen, so + the expectations flip on Windows. All implementations must flip together — + a locator that is absolute to one of them and relative to another is the + same "which binary did we measure?" ambiguity under a different name. + """ + check = "locator-shapes" + win = os.name == "nt" + shapes = [ + # (locator, is it rejected FOR ABSOLUTENESS here?, what it is) + (f".{os.sep}nope-own-cli", True, "explicitly relative"), + (os.path.join("rust", "nope-own-cli"), True, "relative, no leading dot"), + ("C:nope-own-cli.exe", True, "drive-RELATIVE: the drive's current directory"), + ("\\nope\\own-cli.exe", True, "root-relative: the current drive"), + ("C:/nope/own-cli.exe", not win, "drive-rooted, forward slashes"), + ("C:\\nope\\own-cli.exe", not win, "drive-rooted, backslashes"), + ("\\\\.\\C:\\nope\\own-cli.exe", not win, "UNC/device-rooted"), + (str(ROOT / "no-such-own-cli"), False, "this platform's own absolute form"), + ] + problems = [] + for locator, rejected_for_absoluteness, what in shapes: + env = dict(os.environ) + env["OWEN_RUST_CORE"] = locator + seen = {} + r = subprocess.run( + [bash_exe(), str(ROOT / "scripts/own-check.sh"), + "--engine", "rust", "--", str(sample)], + capture_output=True, env=env, cwd=str(ROOT), check=False) + seen["own-check.sh"] = "is not an absolute path" in (r.stdout + r.stderr).decode( + "utf-8", "replace") + dll = launcher_dll() + if dll is not None and have_dotnet(): + r2 = subprocess.run( + ["dotnet", dll, "check", "--engine", "rust", str(sample)], + capture_output=True, env=env, cwd=str(ROOT), check=False) + seen["owen"] = "is not an absolute path" in (r2.stdout + r2.stderr).decode( + "utf-8", "replace") + for surface, got in seen.items(): + if got != rejected_for_absoluteness: + verdict = "refused it as not absolute" if got else "accepted its shape" + problems.append( + f"{surface}: '{locator}' ({what}) — {verdict}, expected the opposite " + f"on {'Windows' if win else 'this POSIX host'}") + if len(set(seen.values())) > 1: + problems.append(f"'{locator}' ({what}) is absolute to one implementation and " + f"relative to another: {seen}") + if problems: + fail(check, "; ".join(problems)) + else: + ok(check, f"{len(shapes)} locator shapes are classified as this platform defines them, " + "and every implementation available here agrees") + + def control_compare_failure_is_classified(sample: Path, tmp: Path) -> None: """D4.1: the compare verdict is what the case WAS, not what a Rust-child field happened to be. @@ -1091,21 +1162,49 @@ def run() -> int: # No fail-fast: every control runs, so a campaign sees every catcher a # mutation trips rather than only the first. - control_bad_locator_is_2(sample_dir, tmp) - control_absolute_locator_only(sample_dir, tmp) - control_compare_failure_is_classified(sample_dir, tmp) - control_no_selector_in_own_cli() - control_default_stays_python(sample_dir) - control_rust_actually_runs_rust(sample_dir) - control_rust_failure_no_fallback(sample_dir) - control_rc70_is_not_a_verdict(sample_dir) - control_unexpected_rc_and_raw_retention(sample_dir, tmp) - control_compare_same_input_and_extract_once(sample_dir, tmp) - control_compare_zero_document(tmp) - control_compare_failure_and_divergence(sample_dir, tmp) - control_candidate_identity(sample_dir, tmp) + # + # OWEN_STAGE1_ONLY names the controls to run, comma-separated. It + # exists for one job: the Windows-native mutation leg evaluates + # mutants whose contract only Windows can be asked about, and running + # the whole suite there would mean building the .NET launcher and the + # fault-injection core for every mutant. It NARROWS the set and can + # never widen it, and an unknown name is an error rather than a + # silently empty run — a filter that quietly selects nothing is a way + # to report a green campaign that measured nothing. + only = [n.strip() for n in os.environ.get("OWEN_STAGE1_ONLY", "").split(",") if n.strip()] + controls = { + "bad-locator-is-2": lambda: control_bad_locator_is_2(sample_dir, tmp), + "absolute-locator-only": lambda: control_absolute_locator_only(sample_dir, tmp), + "locator-shapes": lambda: control_locator_shapes(sample_dir), + "compare-failure-classified": + lambda: control_compare_failure_is_classified(sample_dir, tmp), + "no-selector-in-own-cli": control_no_selector_in_own_cli, + "default-stays-python": lambda: control_default_stays_python(sample_dir), + "rust-actually-runs-rust": lambda: control_rust_actually_runs_rust(sample_dir), + "rust-failure-no-fallback": lambda: control_rust_failure_no_fallback(sample_dir), + "rc70-is-not-a-verdict": lambda: control_rc70_is_not_a_verdict(sample_dir), + "unexpected-rc": + lambda: control_unexpected_rc_and_raw_retention(sample_dir, tmp), + "compare-extracts-once": + lambda: control_compare_same_input_and_extract_once(sample_dir, tmp), + "compare-zero-document": lambda: control_compare_zero_document(tmp), + "divergence-is-5": + lambda: control_compare_failure_and_divergence(sample_dir, tmp), + "candidate-identity": lambda: control_candidate_identity(sample_dir, tmp), + } + unknown = [n for n in only if n not in controls] + if unknown: + print(f"OWEN_STAGE1_ONLY names no such control(s): {', '.join(unknown)}; " + f"known: {', '.join(sorted(controls))}", file=sys.stderr) + return 2 + for name, fn in controls.items(): + if only and name not in only: + continue + fn() print() + if only: + print(f"(OWEN_STAGE1_ONLY narrowed this run to: {', '.join(only)})") print(f"stage-1 engine controls: {len(_PASSES)} passed, " f"{len(_FAILURES)} failed, {len(_SKIPS)} skipped, " f"{len(_NOT_APPLICABLE)} not applicable on this platform") diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py index 869abd3c..cda47c6e 100644 --- a/tests/test_stage1_ps1.py +++ b/tests/test_stage1_ps1.py @@ -26,7 +26,7 @@ mutation whose target is `scripts/own-check.ps1` is only `caught` when a WINDOWS PowerShell catcher observes the mutant and fails. The campaign that owns these mutants therefore runs on a Windows runner -(`.github/workflows/ci.yml`, the `stage1-ps1-mutations` job), and a Linux run +(`.github/workflows/ci.yml`, the `stage1-windows-mutations` job), and a Linux run of this file is a developer convenience, never the record. Failures print `FAIL[]: `; nothing stops at the first one. @@ -176,10 +176,42 @@ def control_absolute_locator(sample: Path, tmp: Path) -> None: problems.append(f"{engine}: refused without naming the absolute requirement") if b"OWN001" in r.stdout: problems.append(f"{engine}: produced a verdict for a relative locator") + + # And the direction a "reject the relative one" assertion cannot see: an + # absolute locator must be ACCEPTED. own-check.sh shipped a validator that + # refused every drive-rooted path and still passed the negative half of + # this check on Linux. own-check.ps1 delegates to IsPathFullyQualified, so + # the same failure would look identical from outside; the assertion is on + # the REASON, and every path here is absent so each run stops at the same + # preflight without a spawn. + win = os.name == "nt" + shapes = [ + (f".{os.sep}nope-own-cli", True, "explicitly relative"), + ("C:nope-own-cli.exe", True, "drive-RELATIVE: the drive's current directory"), + ("\\nope\\own-cli.exe", True, "root-relative: the current drive"), + ("C:/nope/own-cli.exe", not win, "drive-rooted, forward slashes"), + ("C:\\nope\\own-cli.exe", not win, "drive-rooted, backslashes"), + ("\\\\.\\C:\\nope\\own-cli.exe", not win, "UNC/device-rooted"), + (str(ROOT / "no-such-own-cli"), False, "this platform's own absolute form"), + ] + for locator, rejected_for_absoluteness, what in shapes: + r = run_ps1(["-Engine", "rust", "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": locator}) + if r is None: + skip(check, "no pwsh") + return + merged = (r.stdout + r.stderr).decode("utf-8", "replace") + got = "is not an absolute path" in merged + if got != rejected_for_absoluteness: + verdict = "refused it as not absolute" if got else "accepted its shape" + problems.append(f"'{locator}' ({what}) — {verdict}, expected the opposite " + f"on {'Windows' if win else 'this POSIX host'}") + if problems: fail(check, "; ".join(problems)) else: - ok(check, "a relative but existing locator is refused with exit 2") + ok(check, "a relative but existing locator is refused with exit 2, and the absolute " + "shapes this platform defines are accepted") def control_not_started_is_2(sample: Path, tmp: Path) -> None: From d4b55bdb110a8dc97971fb865e9e54b040db278b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:40:22 +0000 Subject: [PATCH 17/22] fix(stage1): own-check.ps1 asked the platform to OPEN the candidate, never to run it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two platforms answered the same question wrong, and both times the platform was not at fault. On Linux, `& $rustCore` against a file the loader cannot start returned 0 and printed `xdg-open: no method available for opening ...`. I read that as a platform limit and declared the spawn-seam control not-applicable off Windows. Windows CI then returned 0 with both streams empty, and the job's own cleanup line terminated an orphaned NOTEPAD. Same defect, two desktop handlers. PowerShell's call operator does not spawn a program; it asks the platform to OPEN it, and a file that is not a runnable image goes to whatever is registered for it. So D3.1's seam could not be reached anywhere: the try/catch added for review defect 3 was unreachable, and `own-check.ps1 -Engine rust` with an existing but unrunnable OWEN_RUST_CORE exited 0 with no findings — a clean bill of health for an analysis that never happened, which is worse than any wrong exit code. `Invoke-CandidateProcess` starts the image with UseShellExecute = $false and nothing redirected, so the child inherits this process's streams and its output still arrives live. The start either succeeds or throws, and the throw is what the existing catch was written for. Measured on Linux: exit 2 with the "could not be started" diagnostic, where the same input previously exited 0. The compare path was already correct — it goes through Invoke-CapturedProcess, which has always set UseShellExecute = $false. Only the direct Rust path opened its candidate. Consequences for the evidence, both of which cut against my earlier reading: The control is required on BOTH platforms and the not-applicable is gone. The seam is answerable anywhere now, because it is a real spawn. The N/A I added one commit ago was describing a bug in this script, not a property of Linux — and it was reasoned from a measurement I trusted too quickly. P07 is the mutation, and it is the shipped code: back to the call operator. It belongs in the Windows campaign with the rest of this surface, where a PowerShell catcher observes it natively. Also kept: the near-miss. This control PASSED on a developer container with no xdg-open installed, because the invocation failed there and looked exactly like a refusal. A verdict that turns on which desktop helper happens to be installed is not measuring the contract. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-windows.json | 13 +++++++- scripts/own-check.ps1 | 45 +++++++++++++++++++++++--- tests/test_stage1_ps1.py | 36 ++++++++++----------- 3 files changed, 70 insertions(+), 24 deletions(-) diff --git a/docs/evidence/p022-stage1-windows.json b/docs/evidence/p022-stage1-windows.json index a1eabeeb..41422384 100644 --- a/docs/evidence/p022-stage1-windows.json +++ b/docs/evidence/p022-stage1-windows.json @@ -2,7 +2,7 @@ "schema": 1, "comment": "GENERATED-BY-HAND definition; the RESULT beside it is recorded by scripts/mutate_campaign.py --run on a WINDOWS runner and the counts are derived from it, never typed.", "campaign": "p022-stage1-windows", - "description": "#262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. Every mutation here is a plausible MISREADING rather than a syntactic accident.", + "description": "#262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. It also carries the spawn seam itself: a candidate must be STARTED, never handed to the platform to open, and the two desktop handlers that proved this — notepad and xdg-open — are why the seam was unreachable on either platform until it was fixed. Every mutation here is a plausible MISREADING rather than a syntactic accident.", "layers": [ { "id": "ps1", @@ -97,6 +97,17 @@ "expected_catchers": [ "ps1::ps1-absolute-locator" ] + }, + { + "id": "P07", + "rule": "the-candidate-is-spawned-not-opened", + "description": "own-check.ps1 goes back to invoking the candidate with the call operator — 'PowerShell runs it either way, why the ceremony?'. It does not run it: it asks the platform to OPEN it, so a file the loader cannot start is handed to a desktop handler (notepad on Windows, xdg-open on Linux), the run exits 0 with empty streams, and Owen reports a clean finding-free analysis of nothing", + "target": "scripts/own-check.ps1", + "pattern": " \\$psi\\.UseShellExecute = \\$false\n \\$proc = \\[System\\.Diagnostics\\.Process\\]::Start\\(\\$psi\\)\n \\$proc\\.WaitForExit\\(\\)\n return \\$proc\\.ExitCode", + "replacement": " & $FilePath @ArgumentList\n return $LASTEXITCODE", + "expected_catchers": [ + "ps1::ps1-not-started-is-2" + ] } ] } diff --git a/scripts/own-check.ps1 b/scripts/own-check.ps1 index e6669700..66caebe8 100644 --- a/scripts/own-check.ps1 +++ b/scripts/own-check.ps1 @@ -80,6 +80,43 @@ param( Set-StrictMode -Version Latest $ErrorActionPreference = "Stop" +function Invoke-CandidateProcess { + <# + .SYNOPSIS + Run the candidate with the parent's streams, as a REAL process spawn. + + .DESCRIPTION + PowerShell's call operator does not spawn a candidate; it asks the + PLATFORM to "open" it, and a file the loader cannot run is then handed to + whatever is registered for it. Measured on both platforms: on the Windows + CI runner a non-image candidate opened in NOTEPAD (the job's own cleanup + terminated it) and `& $rustCore` returned 0 with both streams empty; on + Linux the same file went to xdg-open. Either way own-check.ps1 reported a + clean, finding-free run having analysed nothing — a false "no findings", + which is worse than any exit code. + + UseShellExecute = $false is what makes this a spawn: the image is started + or the start FAILS, with no file association anywhere in the path. The + failure is deliberately allowed to propagate so the caller can map it to + D3.1's configuration exit (2), which is the seam this whole path exists + to honour. + + Nothing is redirected, so the child inherits this process's stdout and + stderr and its output streams live, exactly as the call operator's did. + #> + param( + [Parameter(Mandatory = $true)][string]$FilePath, + [Parameter(Mandatory = $true)][string[]]$ArgumentList + ) + $psi = New-Object System.Diagnostics.ProcessStartInfo + $psi.FileName = $FilePath + foreach ($a in $ArgumentList) { $psi.ArgumentList.Add($a) } + $psi.UseShellExecute = $false + $proc = [System.Diagnostics.Process]::Start($psi) + $proc.WaitForExit() + return $proc.ExitCode +} + function Invoke-CapturedProcess { <# .SYNOPSIS @@ -212,15 +249,15 @@ try { # The PRODUCTION Rust executable, never own-shadow-engine. $rustArgs = @("ownir") + $ownirArgs try { - & $rustCore @rustArgs - $rc = $LASTEXITCODE + $rc = Invoke-CandidateProcess -FilePath $rustCore -ArgumentList $rustArgs } catch { # D3.1's seam: the candidate never STARTED — an existing file the # loader will not run. That is "cannot select the candidate", so it # is a configuration error (2), not Owen failing internally (5). - # On Windows this is the only point at which a non-runnable - # candidate can be detected, since there is no execute bit to test. + # Reaching this catch is why the call above is a spawn and not the + # call operator: an "open" succeeds on a file that cannot run, and + # a seam nothing can ever arrive at is not a seam. [Console]::Error.WriteLine(("own-check: the candidate ``own-cli`` binary could not be started: " + "'$rustCore' ($($_.Exception.Message)). Set OWEN_RUST_CORE to a runnable ``own-cli`` " + "executable. Owen did not fall back to Python.")) diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py index cda47c6e..d8b95bd6 100644 --- a/tests/test_stage1_ps1.py +++ b/tests/test_stage1_ps1.py @@ -220,27 +220,25 @@ def control_not_started_is_2(sample: Path, tmp: Path) -> None: This is the case the CI smoke step could not reach: it used a NONEXISTENT path, which `Test-Path` rejects long before any spawn. + + It took two wrong answers to get here, and both were the production code + rather than the platform. I first declared this control Linux-N/A because + PowerShell there returned 0 and printed `xdg-open: no method available for + opening ...`; Windows CI then returned 0 with both streams empty, and the + job's own cleanup terminated an orphaned NOTEPAD. Same defect, two desktop + handlers: own-check.ps1 was ASKING THE PLATFORM TO OPEN the candidate + rather than spawning it, so the seam could not be reached anywhere and + Owen reported a clean, finding-free run having analysed nothing. With a + real spawn (UseShellExecute = $false) the start either succeeds or throws, + and the contract is answerable on both platforms — so this control is + required on both, and the N/A is gone. + + The near-miss is worth keeping: on a developer container with no xdg-open + installed, this control PASSED, because the invocation failed and looked + exactly like a refusal. A verdict that turns on which desktop helper + happens to be installed is not measuring the contract. """ check = "ps1-not-started-is-2" - if os.name != "nt": - # Measured on a Linux runner: PowerShell there does not refuse a - # non-executable file, it hands it to the DESKTOP OPENER — the run - # exits 0 with `xdg-open: no method available for opening ...`. So - # "the loader will not start this image" is not a state Linux can be - # asked about; it answers a different question and answers it - # successfully. - # - # Worth recording why this was nearly missed: the control PASSED on a - # developer container, for the wrong reason — no xdg-open was - # installed there, so the invocation failed and looked like a refusal. - # A control whose verdict depends on whether a desktop helper happens - # to be present is not measuring the contract. This is exactly the - # case the Windows-native mutation leg exists for. - not_applicable(check, "PowerShell on Linux routes a non-executable file to the desktop " - "opener instead of refusing it, so the spawn seam cannot be posed " - "here; it is required on Windows, where the ps1 mutation campaign " - "runs") - return unstartable = tmp / "not-a-program.txt" unstartable.write_text("this is text, not an executable image\n", encoding="utf-8") From d97102ed21a122a0eec32549e243186b2e25bf20 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 06:52:00 +0000 Subject: [PATCH 18/22] fix(stage1): a control that raises is one failed check, not a lost suite The Windows leg aborted mid-suite with `OSError: [WinError 193] %1 is not a valid Win32 application`, and both faults behind it are mine. `own-check.sh` was handed straight to CreateProcess. On Linux the shebang carries that; Windows has no such thing, and a .sh file is not a Win32 image. The shared harness has had `bash_exe()` for this since the WSL `bash.exe` shim cost a round; the PowerShell harness was written without it. It now has the same helper, refusing System32's WSL launcher for the same reason. The larger fault is what the crash did to the evidence. An unhandled exception took the whole suite down, so the two controls after it never ran and the mutation campaign saw one nameless failure where it needed a named catcher. No-fail-fast has to survive a control that RAISES, not only one that reports. Both harnesses now record an unexpected exception as that check's failure and carry on. The campaign runner names them too. A voided run printed "caught (1 failing test(s))" and stopped, and finding which check that was took two CI rounds while the runner had the name in hand the whole time. It now prints every catcher and names them in the error. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- scripts/mutate_campaign.py | 10 +++++++- tests/test_stage1_engine.py | 8 +++++- tests/test_stage1_ps1.py | 51 +++++++++++++++++++++++++++++++++---- 3 files changed, 62 insertions(+), 7 deletions(-) diff --git a/scripts/mutate_campaign.py b/scripts/mutate_campaign.py index 75e654b6..e6adeb62 100644 --- a/scripts/mutate_campaign.py +++ b/scripts/mutate_campaign.py @@ -863,8 +863,16 @@ def restore() -> None: control = Outcome(definition.control_id, outcome, tuple(catchers), round(time.monotonic() - t0, 1), detail) print(f" -> {outcome} ({len(catchers)} failing test(s))", flush=True) + for c in catchers: + print(f" {c}", flush=True) if outcome != "survived": - raise CampaignError(f"the unmutated tree did not pass ({outcome}): the run is void") + # Name them here. A void run is the one message a reader cannot act on + # without the names: "1 failing test" sent two CI rounds looking for + # which check it was, and the runner knew all along. + raise CampaignError( + f"the unmutated tree did not pass ({outcome}): the run is void" + + (f" — failing: {', '.join(catchers)}" if catchers else "") + + (f" [{detail}]" if detail else "")) assert_tree_unchanged(baseline, "during the control run") outcomes: list[Outcome] = [] diff --git a/tests/test_stage1_engine.py b/tests/test_stage1_engine.py index 67a0a985..99d8bc53 100644 --- a/tests/test_stage1_engine.py +++ b/tests/test_stage1_engine.py @@ -1200,7 +1200,13 @@ def run() -> int: for name, fn in controls.items(): if only and name not in only: continue - fn() + try: + fn() + except Exception as exc: # a raise is this check's failure, not the suite's + # No-fail-fast has to survive a control that RAISES too. An + # abort costs every control after it, and a campaign then reads + # one nameless failure where it needed a named catcher. + fail(name, f"the control itself raised {type(exc).__name__}: {exc}") print() if only: diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py index d8b95bd6..eaed1e20 100644 --- a/tests/test_stage1_ps1.py +++ b/tests/test_stage1_ps1.py @@ -130,6 +130,34 @@ def stub_exe(tmp: Path) -> str | None: return str(out) if r.returncode == 0 and out.is_file() else None +def bash_exe() -> str: + """The bash that can actually run `own-check.sh`. + + Two things this must survive on a Windows runner. `own-check.sh` cannot be + handed to CreateProcess — a .sh file is not a Win32 image, and Windows CI + raised exactly that (WinError 193) where the shebang had quietly carried it + on Linux. And `bash` on PATH there is C:\\Windows\\System32\\bash.exe, the WSL + launcher rather than a shell: with no distribution installed it exits 1 + with a UTF-16 message about installing one, which arrives as a + plausible-looking script failure and is nothing of the kind. + + A harness concern, not a product one: a Windows user runs own-check.sh from + a git-bash prompt, where `bash` is already the right one. + """ + if os.name != "nt": + return "bash" + candidates = [ + os.environ.get("SHELL"), + r"C:\Program Files\Git\bin\bash.exe", + r"C:\Program Files\Git\usr\bin\bash.exe", + shutil.which("bash"), + ] + for cand in candidates: + if cand and "system32" not in cand.lower() and Path(cand).is_file(): + return cand + return "bash" + + def run_ps1(args: list[str], env: dict[str, str] | None = None, cwd: str | None = None) -> subprocess.CompletedProcess[bytes] | None: """Drive own-check.ps1, capturing RAW bytes — the replay contract is about @@ -288,7 +316,8 @@ def control_agreement_replays_bytes(sample: Path, tmp: Path) -> None: # Extract once, then ask the reference what it says about those facts. facts = tmp / "agree.facts.json" ex = subprocess.run( - [str(ROOT / "scripts/own-check.sh"), "--emit-facts", str(facts), "--", str(sample)], + [bash_exe(), str(ROOT / "scripts/own-check.sh"), + "--emit-facts", str(facts), "--", str(sample)], capture_output=True, check=False, cwd=str(ROOT), env={**os.environ, "PYTHONPATH": str(ROOT)}) if not facts.is_file(): @@ -389,10 +418,22 @@ def run() -> int: sample_dir.mkdir() (sample_dir / "Leak.cs").write_text(SAMPLE_CS, encoding="utf-8") - control_absolute_locator(sample_dir, tmp) - control_not_started_is_2(sample_dir, tmp) - control_agreement_replays_bytes(sample_dir, tmp) - control_failure_evidence_exists(sample_dir, tmp) + # No fail-fast, and that has to survive a control that RAISES. A + # harness bug here aborted the suite on Windows (own-check.sh handed + # straight to CreateProcess), so the two controls after it never ran + # and the campaign saw one nameless failure instead of a named check. + # An unexpected exception is this check's failure, reported like any + # other, and the rest of the suite still runs. + for name, control in ( + ("ps1-absolute-locator", control_absolute_locator), + ("ps1-not-started-is-2", control_not_started_is_2), + ("ps1-agreement-replays", control_agreement_replays_bytes), + ("ps1-failure-evidence", control_failure_evidence_exists), + ): + try: + control(sample_dir, tmp) + except Exception as exc: # a raise is this check's failure, not the suite's + fail(name, f"the control itself raised {type(exc).__name__}: {exc}") print() print(f"stage-1 ps1 controls: {len(_PASSES)} passed, {len(_FAILURES)} failed, " From 1d649e99d9c9a8fac0913ac59fcbda94499479bb Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 07:03:08 +0000 Subject: [PATCH 19/22] fix(evidence): a campaign definition must hash the same on both platforms MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows-native campaign passed — 7/7 caught, honesty control survived, run on the exact head — and the gate rejected the result anyway: "the recorded result was taken over a different campaign definition". The definition was identical. Its BYTES were not. `core.autocrlf=true` is the Git for Windows default, so the runner checked the JSON out with CRLF, and `mutate_campaign.py` records the sha256 of the file it ran: 79ad1476f219 the same definition, checked out on Linux 3dda13ce16d6 the same definition, checked out on Windows So a run that measured exactly the right tree was filed as evidence for a different one. The gate was right to refuse it; the hash was reading the checkout setting, not the campaign. `docs/evidence/*.json text eol=lf` pins the bytes that are the contract. The existing .gitattributes comment already named this hazard — "byte-sensitive evidence whose exact bytes are the contract" — and declined to act on it because no campaign ran on Windows yet. One does now. The result files get the same treatment from the other side: Python's text mode writes CRLF on Windows, so `write_result` now opens with newline="\n". A result recorded on Windows has to be byte-identical to one recorded on Linux, or the evidence differs from itself by the platform that happened to take it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- .gitattributes | 9 +++++++++ scripts/mutate_campaign.py | 6 +++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/.gitattributes b/.gitattributes index b4e3ed8d..ef087200 100644 --- a/.gitattributes +++ b/.gitattributes @@ -14,3 +14,12 @@ # evidence" as its own hygiene tail; this rule covers executable shell scripts # only and takes no position on that wider question. *.sh text eol=lf + +# Campaign evidence is hashed BY BYTE, so it must not depend on a checkout +# setting. `mutate_campaign.py` records the sha256 of the definition it ran, +# and the gate refuses a result whose definition hash does not match. The +# #262 Stage-1 Windows campaign runs on a windows-latest runner, where +# `core.autocrlf=true` rewrote these files on checkout: the same definition +# hashed 79ad1476f219 on Linux and 3dda13ce16d6 on Windows, and a run that +# measured exactly the right tree was rejected as evidence for another tree. +docs/evidence/*.json text eol=lf diff --git a/scripts/mutate_campaign.py b/scripts/mutate_campaign.py index e6adeb62..86646285 100644 --- a/scripts/mutate_campaign.py +++ b/scripts/mutate_campaign.py @@ -962,7 +962,11 @@ def write_result(result: Result, definition: Definition, path: str) -> None: doc["command"] = "cargo test -p --no-fail-fast, for every workspace member" doc["control"] = _outcome_json(result.control) doc["mutations"] = [_outcome_json(o) for o in result.mutations] - with open(path, "w", encoding="utf-8") as f: + # newline="\n" explicitly: a campaign now runs on a Windows runner, and + # Python's text mode would write CRLF there. A result recorded on Windows + # has to be byte-identical to one recorded on Linux, or the evidence + # differs from itself by the platform that happened to take it. + with open(path, "w", encoding="utf-8", newline="\n") as f: json.dump(doc, f, indent=2, ensure_ascii=False) f.write("\n") From 4ccf68d1fc6999057cf1d7fac14d33c0690e1af1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 07:14:27 +0000 Subject: [PATCH 20/22] evidence(stage1): both campaigns recorded against the shipped tree, partitioned by platform MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Linux, 21 mutations, recorded at 1d649e99d9c9 on a clean tree: 21/21 caught, 0 survived, 0 compile-error, 0 invalid, 0 runner-error, every expected catcher hit, honesty control M00 survived the unmutated tree. Windows-native, 7 mutations, recorded at the same commit on a windows-latest runner and copied from that job's own printed result: 7/7 caught, control survived, every expected catcher hit. Six target scripts/own-check.ps1 and are killed by PowerShell catchers running on Windows; the seventh targets the drive-rooted arm of own-check.sh, which no Linux catcher can reach, and is killed by the shapes layer there. The two are recorded and reported separately because they are not the same measurement. A mutant evaluated where its contract cannot be posed is decorative, so nothing here is added up across platforms. The result taken at 0f51868 (16/16) is superseded twice over: it predates five mutations and describes production files that have since changed. So is every recording between it and this one — each named a tree that no longer exists, which is the one thing a campaign result must not do. Counts in docs/generated/p022-stage1-mutations.md are projected from these two runs by scripts/render_checkpoint_status.py and never typed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 104 ++++++++++++++---- docs/evidence/p022-stage1-windows.result.json | 82 ++++++++++++++ docs/generated/p022-stage1-mutations.md | 54 +++++---- 3 files changed, 201 insertions(+), 39 deletions(-) create mode 100644 docs/evidence/p022-stage1-windows.result.json diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index a182c69a..b2af0616 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -3,10 +3,10 @@ "comment": "Recorded mutation-campaign run (scripts/mutate_campaign.py --run). Raw facts only: outcomes, catchers, provenance. Counts are derived by scripts/render_checkpoint_status.py; regenerate this file by re-running the campaign, never by hand.", "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", - "definition_sha256": "b27b905145eec08d287830830f0d2f8d7611e1f5345e880b881c7a50d994e3e7", - "source_commit": "0f51868df54041b8a8ed2af1c8b0ce41838009c8", + "definition_sha256": "c8bd2180f285a130dfadc153e966e48b2c2d8d6072d90651906112719c0040eb", + "source_commit": "1d649e99d9c9a8fac0913ac59fcbda94499479bb", "dirty": false, - "recorded_at": "2026-09-09T03:07:20Z", + "recorded_at": "2026-09-09T07:11:44Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 48.9 + "elapsed_seconds": 27.8 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 30.1 + "elapsed_seconds": 24.1 }, { "id": "M02", @@ -32,7 +32,7 @@ "catchers": [ "stage1::rust-actually-runs-rust" ], - "elapsed_seconds": 27.1 + "elapsed_seconds": 22.5 }, { "id": "M03", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 28.7 + "elapsed_seconds": 23.5 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 27.8 + "elapsed_seconds": 23.6 }, { "id": "M05", @@ -60,23 +60,25 @@ "catchers": [ "stage1::raw-rc-retained" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 23.5 }, { "id": "M06", "outcome": "caught", "catchers": [ + "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 28.0 + "elapsed_seconds": 23.8 }, { "id": "M07", "outcome": "caught", "catchers": [ + "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 27.6 + "elapsed_seconds": 23.5 }, { "id": "M08", @@ -84,15 +86,16 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 27.6 + "elapsed_seconds": 23.4 }, { "id": "M09", "outcome": "caught", "catchers": [ + "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 27.5 + "elapsed_seconds": 23.3 }, { "id": "M10", @@ -102,15 +105,16 @@ "stage1::compare-same-input", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.0 + "elapsed_seconds": 23.4 }, { "id": "M11", "outcome": "caught", "catchers": [ + "stage1::compare-failure-classified", "stage1::exec-failure-is-5" ], - "elapsed_seconds": 27.7 + "elapsed_seconds": 22.9 }, { "id": "M12", @@ -118,7 +122,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.3 + "elapsed_seconds": 22.9 }, { "id": "M13", @@ -126,7 +130,7 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 28.1 + "elapsed_seconds": 22.8 }, { "id": "M14", @@ -135,7 +139,7 @@ "stage1::compare-no-substitution", "stage1::divergence-is-5" ], - "elapsed_seconds": 28.9 + "elapsed_seconds": 22.6 }, { "id": "M15", @@ -143,7 +147,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 28.4 + "elapsed_seconds": 23.6 }, { "id": "M16", @@ -151,7 +155,67 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 28.5 + "elapsed_seconds": 23.6 + }, + { + "id": "M17", + "outcome": "caught", + "catchers": [ + "stage1::absolute-locator-only", + "stage1::locator-shapes" + ], + "elapsed_seconds": 25.5 + }, + { + "id": "M18", + "outcome": "caught", + "catchers": [ + "stage1::absolute-locator-only", + "stage1::locator-shapes" + ], + "elapsed_seconds": 28.4 + }, + { + "id": "M19", + "outcome": "caught", + "catchers": [ + "stage1::compare-failure-classified" + ], + "elapsed_seconds": 23.3 + }, + { + "id": "M20", + "outcome": "caught", + "catchers": [ + "stage1::compare-extracts-once", + "stage1::compare-same-input", + "stage1::compare-zero-document", + "stage1::divergence-is-5", + "stage1::exec-failure-is-5", + "stage1::locator-shapes", + "stage1::rust-failure-no-fallback" + ], + "elapsed_seconds": 11.5 + }, + { + "id": "M21", + "outcome": "caught", + "catchers": [ + "stage1::absolute-locator-only", + "stage1::candidate-identity", + "stage1::compare-failure-classified", + "stage1::compare-same-input", + "stage1::compare-zero-document", + "stage1::divergence-is-5", + "stage1::exec-failure-is-5", + "stage1::locator-shapes", + "stage1::raw-rc-retained", + "stage1::rc70-is-not-a-verdict", + "stage1::rust-actually-runs-rust", + "stage1::rust-failure-no-fallback", + "stage1::unexpected-rc-maps-to-5" + ], + "elapsed_seconds": 15.5 } ] } diff --git a/docs/evidence/p022-stage1-windows.result.json b/docs/evidence/p022-stage1-windows.result.json new file mode 100644 index 00000000..718b78f8 --- /dev/null +++ b/docs/evidence/p022-stage1-windows.result.json @@ -0,0 +1,82 @@ +{ + "schema": 1, + "comment": "Recorded mutation-campaign run (scripts/mutate_campaign.py --run). Raw facts only: outcomes, catchers, provenance. Counts are derived by scripts/render_checkpoint_status.py; regenerate this file by re-running the campaign, never by hand.", + "campaign": "p022-stage1-windows", + "definition": "docs/evidence/p022-stage1-windows.json", + "definition_sha256": "79ad1476f219239ecd4753d08dd8fd3ca018e17f35da368ba464196d1428aca0", + "source_commit": "1d649e99d9c9a8fac0913ac59fcbda94499479bb", + "dirty": false, + "recorded_at": "2026-09-09T07:09:15Z", + "layers": [ + "ps1", + "shapes" + ], + "command": "every layer the definition declares, for every mutation", + "control": { + "id": "M00", + "outcome": "survived", + "catchers": [], + "elapsed_seconds": 66.6 + }, + "mutations": [ + { + "id": "P01", + "outcome": "caught", + "catchers": [ + "ps1::ps1-absolute-locator" + ], + "elapsed_seconds": 34.4 + }, + { + "id": "P02", + "outcome": "caught", + "catchers": [ + "ps1::ps1-not-started-is-2" + ], + "elapsed_seconds": 26.4 + }, + { + "id": "P03", + "outcome": "caught", + "catchers": [ + "ps1::ps1-agreement-replays" + ], + "elapsed_seconds": 24.1 + }, + { + "id": "P04", + "outcome": "caught", + "catchers": [ + "ps1::ps1-failure-evidence" + ], + "elapsed_seconds": 24.9 + }, + { + "id": "P05", + "outcome": "caught", + "catchers": [ + "shapes::locator-shapes" + ], + "elapsed_seconds": 24.2 + }, + { + "id": "P06", + "outcome": "caught", + "catchers": [ + "ps1::ps1-absolute-locator", + "ps1::ps1-agreement-replays", + "ps1::ps1-failure-evidence", + "ps1::ps1-not-started-is-2" + ], + "elapsed_seconds": 17.0 + }, + { + "id": "P07", + "outcome": "caught", + "catchers": [ + "ps1::ps1-not-started-is-2" + ], + "elapsed_seconds": 22.6 + } + ] +} diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index ebc6a168..6a38261d 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -12,10 +12,10 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 | measure | value | |--------------------------------------------------|---| -| recorded at commit | `0f51868df54041b8a8ed2af1c8b0ce41838009c8` | +| recorded at commit | `1d649e99d9c9a8fac0913ac59fcbda94499479bb` | | layers run (every one, for every mutation) | `stage1` | | mutations | 21 | -| caught | 16 | +| caught | 21 | | survived | 0 | | compile-error (no evidence either way) | 0 | | invalid-mutation | 0 | @@ -23,11 +23,6 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 | caught without every expected catcher | none | | honesty control `M00` (unmutated tree must pass) | survived — as required | -**This run is not evidence:** - -- the recorded result was taken over a different campaign definition (sha256 or campaign name differs) — re-run the campaign -- result/definition mutation sets differ (missing ['M17', 'M18', 'M19', 'M20', 'M21'], unknown []) - | id | rule | mutation | outcome | caught by | |---|---|---|---|---| | M01 | default-is-python | the Stage-1 default engine is Rust — the cutover read as already decided, instead of Python remaining default until Gate G3 | caught | `stage1::default-stays-python` | @@ -35,27 +30,48 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 | M03 | 70-is-not-a-verdict | 70 is treated as a legal engine result — the shared internal-error code mistaken for part of the verdict contract because both engines document it | caught | `stage1::rc70-is-not-a-verdict`
`stage1::rust-failure-no-fallback` | | M04 | unexpected-rc-maps-to-5 | an unexpected Rust child status passes through as itself — 'propagate the child's exit code' read as faithfulness rather than as leaking a meaningless number to the caller | caught | `stage1::raw-rc-retained`
`stage1::rc70-is-not-a-verdict`
`stage1::rust-failure-no-fallback`
`stage1::unexpected-rc-maps-to-5` | | M05 | raw-rc-retained | the raw child status is dropped from the report — the human-readable cause already names the number, so the typed carrier looks redundant | caught | `stage1::raw-rc-retained` | -| M06 | bad-locator-is-2-not-5 | an unusable OWEN_RUST_CORE is an internal error — a failure to start the engine read as Owen's own bug rather than the caller's configuration | caught | `stage1::bad-locator-is-2` | -| M07 | bad-locator-is-2-not-3 | an unusable OWEN_RUST_CORE reuses exit 3 — 'no usable engine runtime' read as the same class as 'no usable Python', which it is not | caught | `stage1::bad-locator-is-2` | +| M06 | bad-locator-is-2-not-5 | an unusable OWEN_RUST_CORE is an internal error — a failure to start the engine read as Owen's own bug rather than the caller's configuration | caught | `stage1::absolute-locator-only`
`stage1::bad-locator-is-2` | +| M07 | bad-locator-is-2-not-3 | an unusable OWEN_RUST_CORE reuses exit 3 — 'no usable engine runtime' read as the same class as 'no usable Python', which it is not | caught | `stage1::absolute-locator-only`
`stage1::bad-locator-is-2` | | M08 | no-fallback-in-the-shell | the shell falls back to Python when the Rust core produces no verdict — 'be helpful, still give the user an answer', which is precisely the silent fallback every ruling forbids | caught | `stage1::rust-failure-no-fallback` | -| M09 | shell-locator-is-2-not-3 | the shell reports an unusable OWEN_RUST_CORE as exit 3 — the Python-specific 'no usable runtime' code borrowed for the Rust candidate | caught | `stage1::bad-locator-is-2` | +| M09 | shell-locator-is-2-not-3 | the shell reports an unusable OWEN_RUST_CORE as exit 3 — the Python-specific 'no usable runtime' code borrowed for the Rust candidate | caught | `stage1::absolute-locator-only`
`stage1::bad-locator-is-2` | | M10 | divergence-is-5 | a divergence exposes the reference's result — 'Python is still the reference, so trust it' read as a licence to answer while the two engines disagree | caught | `stage1::compare-no-substitution`
`stage1::compare-same-input`
`stage1::divergence-is-5` | -| M11 | exec-failure-is-not-agreement | the execution-failure check is skipped — comparing the results first and treating a crashed engine as just another difference, which loses the distinction D4.1 draws between (b) and (c) | caught | `stage1::exec-failure-is-5` | +| M11 | exec-failure-is-not-agreement | the execution-failure check is skipped — comparing the results first and treating a crashed engine as just another difference, which loses the distinction D4.1 draws between (b) and (c) | caught | `stage1::compare-failure-classified`
`stage1::exec-failure-is-5` | | M12 | zero-document-compare-fails | a document with no analysable unit is compared anyway — 'both engines agreed' read as a result rather than as a zero denominator | caught | `stage1::compare-zero-document` | | M13 | candidate-identity-recorded | the evidence records a placeholder candidate digest — the path already names the binary, so hashing it looks like belt-and-braces | caught | `stage1::candidate-identity` | | M14 | shell-compare-checks-stdout | the shell compare stops comparing stdout — the exit code read as the whole of 'the public result', dropping the bytes the user actually sees | caught | `stage1::compare-no-substitution`
`stage1::divergence-is-5` | | M15 | shell-divergence-is-5 | the shell reports a divergence as exit 1 — the 'something is wrong' tier reached for, when in public Owen 1 already means findings | caught | `stage1::divergence-is-5` | | M16 | shell-zero-document-fails | the shell's zero-document guard is dropped — an empty document read as a legitimately clean agreement | caught | `stage1::compare-zero-document` | -| M17 | locator-must-be-absolute | the launcher accepts a relative OWEN_RUST_CORE — `GetFullPath` reads as 'it resolves the path for me', which it does: against whatever directory Owen happened to run in | **not recorded** | — | -| M18 | shell-locator-must-be-absolute | the shell accepts a relative OWEN_RUST_CORE — the -f/-x tests look like they answer 'is this a usable binary', and they do, for whatever the current directory made of the path | **not recorded** | — | -| M19 | compare-verdict-is-stated | the compare verdict is inferred from the Rust-child field again — `child_exit_code is null` reads as 'no engine crashed', but it is only ever about the RUST child, so a Python-only failure is stamped 'divergence' | **not recorded** | — | -| M20 | absolute-locator-is-accepted | the shell's POSIX arm stops recognising an absolute path — the over-rejection direction of D3, where a validator that refuses EVERYTHING passes every 'reject the relative one' assertion and makes the tool unusable with a correct configuration | **not recorded** | — | -| M21 | absolute-locator-is-accepted | the launcher's absoluteness test is inverted — the same over-rejection direction in the C# implementation: every fully qualified locator is refused as 'not absolute' and every relative one is admitted | **not recorded** | — | +| M17 | locator-must-be-absolute | the launcher accepts a relative OWEN_RUST_CORE — `GetFullPath` reads as 'it resolves the path for me', which it does: against whatever directory Owen happened to run in | caught | `stage1::absolute-locator-only`
`stage1::locator-shapes` | +| M18 | shell-locator-must-be-absolute | the shell accepts a relative OWEN_RUST_CORE — the -f/-x tests look like they answer 'is this a usable binary', and they do, for whatever the current directory made of the path | caught | `stage1::absolute-locator-only`
`stage1::locator-shapes` | +| M19 | compare-verdict-is-stated | the compare verdict is inferred from the Rust-child field again — `child_exit_code is null` reads as 'no engine crashed', but it is only ever about the RUST child, so a Python-only failure is stamped 'divergence' | caught | `stage1::compare-failure-classified` | +| M20 | absolute-locator-is-accepted | the shell's POSIX arm stops recognising an absolute path — the over-rejection direction of D3, where a validator that refuses EVERYTHING passes every 'reject the relative one' assertion and makes the tool unusable with a correct configuration | caught | `stage1::compare-extracts-once`
`stage1::compare-same-input`
`stage1::compare-zero-document`
`stage1::divergence-is-5`
`stage1::exec-failure-is-5`
`stage1::locator-shapes`
`stage1::rust-failure-no-fallback` | +| M21 | absolute-locator-is-accepted | the launcher's absoluteness test is inverted — the same over-rejection direction in the C# implementation: every fully qualified locator is refused as 'not absolute' and every relative one is admitted | caught | `stage1::absolute-locator-only`
`stage1::candidate-identity`
`stage1::compare-failure-classified`
`stage1::compare-same-input`
`stage1::compare-zero-document`
`stage1::divergence-is-5`
`stage1::exec-failure-is-5`
`stage1::locator-shapes`
`stage1::raw-rc-retained`
`stage1::rc70-is-not-a-verdict`
`stage1::rust-actually-runs-rust`
`stage1::rust-failure-no-fallback`
`stage1::unexpected-rc-maps-to-5` | ## Stage 1 — the surfaces only Windows can be asked about: `own-check.ps1`, and the drive-rooted arm of the shell's locator classifier. Measured on a WINDOWS runner, because a mutant of either is invisible to a Linux catcher -Campaign `p022-stage1-windows` — #262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. Every mutation here is a plausible MISREADING rather than a syntactic accident. +Campaign `p022-stage1-windows` — #262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. It also carries the spawn seam itself: a candidate must be STARTED, never handed to the platform to open, and the two desktop handlers that proved this — notepad and xdg-open — are why the seam was unreachable on either platform until it was fixed. Every mutation here is a plausible MISREADING rather than a syntactic accident. -Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `eeca9e0d6910c849…`, 6 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. +Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `79ad1476f219239e…`, 7 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. -**No recorded run** is committed (expected at `docs/evidence/p022-stage1-windows.result.json`): the campaign has a definition but no evidence. Nothing below is a number. +| measure | value | +|--------------------------------------------------|---| +| recorded at commit | `1d649e99d9c9a8fac0913ac59fcbda94499479bb` | +| layers run (every one, for every mutation) | `ps1`, `shapes` | +| mutations | 7 | +| caught | 7 | +| survived | 0 | +| compile-error (no evidence either way) | 0 | +| invalid-mutation | 0 | +| runner-error | 0 | +| caught without every expected catcher | none | +| honesty control `M00` (unmutated tree must pass) | survived — as required | + +| id | rule | mutation | outcome | caught by | +|---|---|---|---|---| +| P01 | ps1-locator-must-be-absolute | own-check.ps1 accepts a relative OWEN_RUST_CORE — Test-Path says the file is there, which is true and not the question D3 asks | caught | `ps1::ps1-absolute-locator` | +| P02 | ps1-not-started-is-configuration | own-check.ps1 reports a candidate that never started as an internal failure — 'the engine blew up' read as Owen's bug rather than the caller's configuration, which is the exact side of D3.1's seam the old comment got backwards | caught | `ps1::ps1-not-started-is-2` | +| P03 | ps1-agreement-replays-raw-bytes | own-check.ps1 replays agreement through the object pipeline again, stdout only — `Get-Content | Write-Output` looks like an echo and is a decode-and-re-encode that also drops stderr | caught | `ps1::ps1-agreement-replays` | +| P04 | ps1-failure-evidence-survives | own-check.ps1 deletes the reproduction directory it just named — cleanup reads as tidiness, and the message that pointed at it is left describing something that no longer exists | caught | `ps1::ps1-failure-evidence` | +| P05 | absolute-locator-is-accepted | the shell's drive-rooted arm stops matching — this is the defect that actually shipped: `[/\]` escapes the closing bracket, so the set is unterminated and matches NEITHER `C:/` nor `C:\`, and every correct Windows locator was refused as 'not absolute' while every Linux control stayed green | caught | `shapes::locator-shapes` | +| P06 | absolute-locator-is-accepted | own-check.ps1's absoluteness test is inverted — the over-rejection direction on this surface: every fully qualified locator is refused as 'not absolute' and every relative one is admitted, which no assertion that only feeds it a relative path can see | caught | `ps1::ps1-absolute-locator`
`ps1::ps1-agreement-replays`
`ps1::ps1-failure-evidence`
`ps1::ps1-not-started-is-2` | +| P07 | the-candidate-is-spawned-not-opened | own-check.ps1 goes back to invoking the candidate with the call operator — 'PowerShell runs it either way, why the ceremony?'. It does not run it: it asks the platform to OPEN it, so a file the loader cannot start is handed to a desktop handler (notepad on Windows, xdg-open on Linux), the run exits 0 with empty streams, and Owen reports a clean finding-free analysis of nothing | caught | `ps1::ps1-not-started-is-2` | From da897b42fcd76ea1914d286bd6bae6065c072c70 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 08:26:09 +0000 Subject: [PATCH 21/22] test(stage1): the stderr half of "both streams" was never compared MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two proof gaps and one lie in a comment, all found by review against primary source rather than by anything in this repository. 1. The agreement control's stderr assertion never ran. It was guarded by `if ref.stderr`, and a healthy `python -m ownlang ownir` writes none: CI printed `184 out, 0 err` on Windows and `160 out, 0 err` on Linux while the control reported that it had compared "both streams". A guard that switches an assertion off exactly when the data is ordinary is not a guard, it is a hole with a denominator of zero — and this branch has spent two review rounds on honest denominators. The fixture is now built rather than found. `python` is resolved by own-check.ps1 as a bare command, so a shim directory whose `python` IS the committed native stub makes the REFERENCE's bytes choosable; both engines then run the same stub over the same fixture and genuinely agree, with stderr that is deliberately not empty. The bytes are chosen to survive nothing: CRLF a text pipeline would rewrite, multi-byte UTF-8 a decode-and-re-encode would normalise. stdout is compared exactly; stderr as a suffix, because the extraction ahead of the compare sends dotnet's chatter there and that chatter is not the launcher's replay. The old fixture is kept beside it. It is the only one that proves the replay is faithful to the REAL reference implementation, which a stub cannot show. P03 no longer breaks both streams at once — it broke stdout AND emptied $errBytes, so its death proved only that stdout was checked. It is now scoped to stdout, and P08 removes the stderr replay alone while leaving stdout byte-faithful. Measured separately: P03 dies on stdout in both fixtures with the stderr assertion silent; P08 dies only on stderr, against a launcher that emitted 0 bytes there. "Both streams" is load-bearing now, not grammar. 2. The PowerShell surface never proved D3's preflight POSITION. The shared harness counts extractor invocations, but its counter is a `#!` script installed only off Windows, so on the one platform whose campaign is the record there was no counter at all. A validator that drifted to after extraction would still exit 2, still name the absolute requirement, still emit no verdict, and every assertion in the control would stay green while Owen paid for a full Roslyn pass over a candidate it was about to refuse. There is a counter now on both platforms — a `.cmd` on Windows, since own-check.ps1 reaches dotnet through PowerShell's call operator and that honours PATHEXT — and P09 is the mutant it exists to kill: the toolchain is probed before the locator is validated, which is a thing CLIs really do, and nothing else in the control notices. It counts EXTRACTIONS, not every dotnet invocation, and that distinction was measured, not assumed. A count-everything assertion failed on this development container and would have passed on the Windows runner: where pwsh is installed as a dotnet global tool, merely starting the shell invokes dotnet twice. A control whose verdict turns on how the shell was packaged is not measuring Owen — the same shape of mistake as the xdg-open near-miss earlier in this branch. 3. RustCoreLocator's IsExecutable still documented the opposite of the ratified seam: a Windows broken image "already maps to the internal-error path". It maps to exit 2 — EngineRunner raises RustCoreNotStartedException and CheckCommand returns RustCoreLocator.ExitCode — because a candidate that never started never ran and cannot have misbehaved. The comment below it was wrong for the same reason, claiming a false "yes" would produce an internal error; both answers now reach the same exit code, and what a false "yes" actually costs is the diagnostic and the moment it arrives. A ratified seam described backwards in production source is a defect whether or not it executes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-windows.json | 26 ++- docs/generated/p022-stage1-mutations.md | 13 +- .../roslyn/OwnSharp.Cli/RustCoreLocator.cs | 22 +- tests/test_stage1_ps1.py | 204 +++++++++++++++--- 4 files changed, 223 insertions(+), 42 deletions(-) diff --git a/docs/evidence/p022-stage1-windows.json b/docs/evidence/p022-stage1-windows.json index 41422384..8b656484 100644 --- a/docs/evidence/p022-stage1-windows.json +++ b/docs/evidence/p022-stage1-windows.json @@ -57,10 +57,10 @@ { "id": "P03", "rule": "ps1-agreement-replays-raw-bytes", - "description": "own-check.ps1 replays agreement through the object pipeline again, stdout only — `Get-Content | Write-Output` looks like an echo and is a decode-and-re-encode that also drops stderr", + "description": "own-check.ps1 replays STDOUT through `Get-Content -Raw | Write-Output` — a decode-and-re-encode read as an echo. Scoped to stdout ONLY: it leaves the stderr replay intact, so it can be killed by nothing but the stdout half of the byte-faithfulness assertion. The earlier version of this mutation also emptied $errBytes, which meant its death proved only that stdout was checked and left 'both streams' as grammar", "target": "scripts/own-check.ps1", "pattern": " \\$outBytes = \\[System\\.IO\\.File\\]::ReadAllBytes\\(\\(Join-Path \\$cmpDir \"python\\.out\"\\)\\)\n \\$errBytes = \\[System\\.IO\\.File\\]::ReadAllBytes\\(\\(Join-Path \\$cmpDir \"python\\.err\"\\)\\)", - "replacement": " Get-Content -LiteralPath (Join-Path $cmpDir \"python.out\") -Raw -ErrorAction SilentlyContinue | Write-Output\n $outBytes = @(); $errBytes = @()", + "replacement": " Get-Content -LiteralPath (Join-Path $cmpDir \"python.out\") -Raw -ErrorAction SilentlyContinue | Write-Output\n $outBytes = @()\n $errBytes = [System.IO.File]::ReadAllBytes((Join-Path $cmpDir \"python.err\"))", "expected_catchers": [ "ps1::ps1-agreement-replays" ] @@ -108,6 +108,28 @@ "expected_catchers": [ "ps1::ps1-not-started-is-2" ] + }, + { + "id": "P08", + "rule": "ps1-agreement-replays-raw-bytes", + "description": "own-check.ps1 drops the STDERR half of the agreement replay while leaving stdout byte-faithful — 'stderr is diagnostics, the result is stdout', which is how the original defect was written in the first place. It is the twin of P03 and exists because a single mutation that broke both streams could be killed by the stdout assertion alone: with this one, the stderr assertion is the only thing standing between the mutant and a green run", + "target": "scripts/own-check.ps1", + "pattern": " if \\(\\$errBytes\\.Length -gt 0\\) \\{\n \\$stderrStream = \\[System\\.Console\\]::OpenStandardError\\(\\)\n \\$stderrStream\\.Write\\(\\$errBytes, 0, \\$errBytes\\.Length\\)\n \\$stderrStream\\.Flush\\(\\)\n \\}", + "replacement": " # the result is stdout; stderr is only diagnostics", + "expected_catchers": [ + "ps1::ps1-agreement-replays" + ] + }, + { + "id": "P09", + "rule": "the-locator-is-a-preflight", + "description": "own-check.ps1 checks that the .NET toolchain answers before it validates the locator — 'fail on the missing tool first', which plenty of CLIs do. The rejection it then produces is still correct in every observable way: exit 2, the absolute requirement named, no verdict. What it loses is the POSITION: a decision reachable from one environment variable now happens behind an external process, and on the real path that process is a Roslyn extraction over the caller's whole tree. Nothing in this control caught that until it began counting child processes", + "target": "scripts/own-check.ps1", + "pattern": " if \\(\\$problem -ne \"\"\\) \\{\n # Windows has no execute bit", + "replacement": " if ($problem -ne \"\") {\n & dotnet run --project (Join-Path $Root \"frontend/roslyn/OwnSharp.Extractor\") -- @Paths 1>$null\n # Windows has no execute bit", + "expected_catchers": [ + "ps1::ps1-absolute-locator" + ] } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index 6a38261d..f9ec029c 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -51,13 +51,13 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 Campaign `p022-stage1-windows` — #262 Stage 1 — the surfaces whose contract only Windows can be asked about. `scripts/own-check.ps1` was driven by a CI smoke step but was absent from the adversarial control set, which is why three of its engine-contract defects survived a 16/16 campaign on the other surfaces: a campaign can only prove what some control observes. Beside it sits the Windows half of `scripts/own-check.sh`'s locator classifier, which the Linux campaign cannot reach at all — a drive-rooted path is absolute only where Windows resolves it, and a mutant of that arm is invisible on Linux whatever the runner reports. It also carries the spawn seam itself: a candidate must be STARTED, never handed to the platform to open, and the two desktop handlers that proved this — notepad and xdg-open — are why the seam was unreachable on either platform until it was fixed. Every mutation here is a plausible MISREADING rather than a syntactic accident. -Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `79ad1476f219239e…`, 7 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. +Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `6dbfa6054f6447bb…`, 9 mutations). Replay on a clean tree with `python scripts/mutate_campaign.py --campaign docs/evidence/p022-stage1-windows.json --run`; the recorded run is raw outcomes and provenance, the counts below are derived from it. | measure | value | |--------------------------------------------------|---| | recorded at commit | `1d649e99d9c9a8fac0913ac59fcbda94499479bb` | | layers run (every one, for every mutation) | `ps1`, `shapes` | -| mutations | 7 | +| mutations | 9 | | caught | 7 | | survived | 0 | | compile-error (no evidence either way) | 0 | @@ -66,12 +66,19 @@ Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `79ad1476f219239e | caught without every expected catcher | none | | honesty control `M00` (unmutated tree must pass) | survived — as required | +**This run is not evidence:** + +- the recorded result was taken over a different campaign definition (sha256 or campaign name differs) — re-run the campaign +- result/definition mutation sets differ (missing ['P08', 'P09'], unknown []) + | id | rule | mutation | outcome | caught by | |---|---|---|---|---| | P01 | ps1-locator-must-be-absolute | own-check.ps1 accepts a relative OWEN_RUST_CORE — Test-Path says the file is there, which is true and not the question D3 asks | caught | `ps1::ps1-absolute-locator` | | P02 | ps1-not-started-is-configuration | own-check.ps1 reports a candidate that never started as an internal failure — 'the engine blew up' read as Owen's bug rather than the caller's configuration, which is the exact side of D3.1's seam the old comment got backwards | caught | `ps1::ps1-not-started-is-2` | -| P03 | ps1-agreement-replays-raw-bytes | own-check.ps1 replays agreement through the object pipeline again, stdout only — `Get-Content | Write-Output` looks like an echo and is a decode-and-re-encode that also drops stderr | caught | `ps1::ps1-agreement-replays` | +| P03 | ps1-agreement-replays-raw-bytes | own-check.ps1 replays STDOUT through `Get-Content -Raw | Write-Output` — a decode-and-re-encode read as an echo. Scoped to stdout ONLY: it leaves the stderr replay intact, so it can be killed by nothing but the stdout half of the byte-faithfulness assertion. The earlier version of this mutation also emptied $errBytes, which meant its death proved only that stdout was checked and left 'both streams' as grammar | caught | `ps1::ps1-agreement-replays` | | P04 | ps1-failure-evidence-survives | own-check.ps1 deletes the reproduction directory it just named — cleanup reads as tidiness, and the message that pointed at it is left describing something that no longer exists | caught | `ps1::ps1-failure-evidence` | | P05 | absolute-locator-is-accepted | the shell's drive-rooted arm stops matching — this is the defect that actually shipped: `[/\]` escapes the closing bracket, so the set is unterminated and matches NEITHER `C:/` nor `C:\`, and every correct Windows locator was refused as 'not absolute' while every Linux control stayed green | caught | `shapes::locator-shapes` | | P06 | absolute-locator-is-accepted | own-check.ps1's absoluteness test is inverted — the over-rejection direction on this surface: every fully qualified locator is refused as 'not absolute' and every relative one is admitted, which no assertion that only feeds it a relative path can see | caught | `ps1::ps1-absolute-locator`
`ps1::ps1-agreement-replays`
`ps1::ps1-failure-evidence`
`ps1::ps1-not-started-is-2` | | P07 | the-candidate-is-spawned-not-opened | own-check.ps1 goes back to invoking the candidate with the call operator — 'PowerShell runs it either way, why the ceremony?'. It does not run it: it asks the platform to OPEN it, so a file the loader cannot start is handed to a desktop handler (notepad on Windows, xdg-open on Linux), the run exits 0 with empty streams, and Owen reports a clean finding-free analysis of nothing | caught | `ps1::ps1-not-started-is-2` | +| P08 | ps1-agreement-replays-raw-bytes | own-check.ps1 drops the STDERR half of the agreement replay while leaving stdout byte-faithful — 'stderr is diagnostics, the result is stdout', which is how the original defect was written in the first place. It is the twin of P03 and exists because a single mutation that broke both streams could be killed by the stdout assertion alone: with this one, the stderr assertion is the only thing standing between the mutant and a green run | **not recorded** | — | +| P09 | the-locator-is-a-preflight | own-check.ps1 checks that the .NET toolchain answers before it validates the locator — 'fail on the missing tool first', which plenty of CLIs do. The rejection it then produces is still correct in every observable way: exit 2, the absolute requirement named, no verdict. What it loses is the POSITION: a decision reachable from one environment variable now happens behind an external process, and on the real path that process is a Roslyn extraction over the caller's whole tree. Nothing in this control caught that until it began counting child processes | **not recorded** | — | diff --git a/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs index e740f829..ed9e6d38 100644 --- a/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs +++ b/frontend/roslyn/OwnSharp.Cli/RustCoreLocator.cs @@ -133,9 +133,14 @@ private static string Problem(string what) => /// On Unix that is a real permission question, so it is asked of the /// file mode: any of the three execute bits. On Windows there is no /// execute bit — runnability is decided by the loader — so an existing - /// regular file is accepted and a genuinely broken image fails later, at - /// spawn, which is the D3.1 seam's other side and already maps to the - /// internal-error path. + /// regular file is accepted here and a genuinely broken image is refused + /// later, at spawn. + /// + /// That later refusal is still D3.1's CONFIGURATION side, not the + /// internal-error path: EngineRunner raises RustCoreNotStartedException + /// and CheckCommand maps it to this class's ExitCode. A candidate that + /// never started never ran, so it cannot have misbehaved, and 5 would + /// blame Owen for the caller's setting. /// private static bool IsExecutable(string path) { @@ -153,10 +158,13 @@ private static bool IsExecutable(string path) catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or PlatformNotSupportedException) { - // Cannot read the mode: refuse rather than assume runnable. A - // false "yes" here would turn a configuration error into a spawn - // failure reported as an internal error, which loses D3.1's - // distinction between "could not select" and "ran and misbehaved". + // Cannot read the mode: refuse rather than assume runnable. Both + // answers reach the same exit code — a spawn failure is ExitCode + // too — so what a false "yes" costs is the DIAGNOSTIC and the + // moment it arrives: the caller is told the candidate "could not + // be started" instead of that it is not executable, after Owen has + // already paid for a full extraction. The preflight exists to say + // the true thing before doing the expensive thing. return false; } } diff --git a/tests/test_stage1_ps1.py b/tests/test_stage1_ps1.py index eaed1e20..210f4ea0 100644 --- a/tests/test_stage1_ps1.py +++ b/tests/test_stage1_ps1.py @@ -36,6 +36,7 @@ from __future__ import annotations +import json import os import shutil import subprocess @@ -172,6 +173,49 @@ def run_ps1(args: list[str], env: dict[str, str] | None = None, capture_output=True, env=e, cwd=cwd or str(ROOT), check=False) +def _dotnet_tally_shim(tmp: Path) -> tuple[Path, Path] | None: + """A `dotnet` that records every invocation and then delegates to the real + one, so "the extractor never ran" can be MEASURED rather than assumed. + + D3's locator check is a preflight: a validator that drifted to after + extraction would still exit 2, still name the absolute requirement, and + still emit no verdict — every assertion in the caller would stay green + while Owen had already spent a full Roslyn extraction on a candidate it + was about to refuse. The shared harness counts these invocations, but its + shim is a `#!` script installed only off Windows, so the PowerShell surface + had no counter at all on the one platform whose campaign is the record. + + own-check.ps1 reaches dotnet through PowerShell's call operator, which + resolves through PATHEXT, so a `.cmd` is startable there. (The stub cannot + stand in here: this shim has to pass the real invocation through, not + answer it.) + """ + real = shutil.which("dotnet") + if real is None: + return None + shim = tmp / "ps1-dotnet-shim" + shim.mkdir(exist_ok=True) + tally = tmp / "ps1-dotnet-invocations.log" + if tally.exists(): + tally.unlink() + if os.name == "nt": + (shim / "dotnet.cmd").write_text( + "@echo off\r\n" + f'>>"{tally}" echo %*\r\n' + f'"{real}" %*\r\n' + "exit /b %ERRORLEVEL%\r\n", + encoding="utf-8") + else: + script = shim / "dotnet" + script.write_text( + "#!/usr/bin/env bash\n" + f'printf "%s\\n" "$*" >> {json.dumps(str(tally))}\n' + f'exec {json.dumps(real)} "$@"\n', + encoding="utf-8") + script.chmod(0o755) + return shim, tally + + # --- controls -------------------------------------------------------------- @@ -191,9 +235,13 @@ def control_absolute_locator(sample: Path, tmp: Path) -> None: local.chmod(0o755) problems = [] + counted = _dotnet_tally_shim(tmp) for engine in ("rust", "compare"): + env = {"OWEN_RUST_CORE": f".{os.sep}{local.name}"} + if counted is not None: + env["PATH"] = f"{counted[0]}{os.pathsep}{os.environ.get('PATH', '')}" r = run_ps1(["-Engine", engine, "-Format", "human", str(sample)], - env={"OWEN_RUST_CORE": f".{os.sep}{local.name}"}, cwd=str(workdir)) + env=env, cwd=str(workdir)) if r is None: skip(check, "no pwsh") return @@ -205,6 +253,29 @@ def control_absolute_locator(sample: Path, tmp: Path) -> None: if b"OWN001" in r.stdout: problems.append(f"{engine}: produced a verdict for a relative locator") + # The preflight assertion, over both engine paths at once: the EXTRACTOR + # never ran. A validator that drifted to after extraction still exits 2, + # still names the absolute requirement and still emits no verdict, so every + # other assertion above stays green while Owen spends a full Roslyn pass + # over the caller's tree on a candidate it is about to refuse. + # + # It counts extractor invocations, not every dotnet invocation, and the + # difference was measured rather than assumed: where pwsh is installed as a + # dotnet global tool — this container, for one — merely STARTING the shell + # invokes dotnet twice, so a count-everything assertion fails there and + # passes on a Windows runner where pwsh is a real executable. A control + # whose verdict turns on how the shell was packaged is not measuring Owen. + if counted is None: + problems.append("no dotnet on PATH, so 'the extractor never ran' could not be measured — " + "this control's preflight half needs a counter, not an assumption") + else: + lines = counted[1].read_text(encoding="utf-8").splitlines() if counted[1].exists() else [] + extractions = [ln for ln in lines if "OwnSharp.Extractor" in ln] + if extractions: + problems.append(f"the extractor ran {len(extractions)} time(s) before the locator was " + f"rejected — validation drifted past the preflight " + f"[{extractions[0][:100]}]") + # And the direction a "reject the relative one" assertion cannot see: an # absolute locator must be ACCEPTED. own-check.sh shipped a validator that # refused every drive-rooted path and still passed the negative half of @@ -238,8 +309,9 @@ def control_absolute_locator(sample: Path, tmp: Path) -> None: if problems: fail(check, "; ".join(problems)) else: - ok(check, "a relative but existing locator is refused with exit 2, and the absolute " - "shapes this platform defines are accepted") + ok(check, "a relative but existing locator is refused with exit 2 on both engine paths " + "before any extraction, and the absolute shapes this platform defines are " + "accepted") def control_not_started_is_2(sample: Path, tmp: Path) -> None: @@ -293,16 +365,57 @@ def control_not_started_is_2(sample: Path, tmp: Path) -> None: ok(check, "an existing-but-unstartable candidate is exit 2 on both engine paths") +def _stub_as_python(tmp: Path, stub: str) -> Path: + """A directory whose `python` IS the controllable stub, for PATH. + + own-check.ps1 runs the reference as the literal command `python`; there is + no OWEN_PYTHON on this surface to redirect. Putting the stub here under + that name is what makes the REFERENCE's bytes choosable, which is the only + way to reach an agreement whose stderr is non-empty. A real `python -m + ownlang ownir` run over a healthy input writes nothing to stderr — measured + on both CI platforms, which reported `0 err` — so a fixture built from it + can never exercise the stderr half of the replay. + """ + shim = tmp / "ps1-python-shim" + shim.mkdir(exist_ok=True) + dest = shim / ("python.exe" if os.name == "nt" else "python") + if not dest.is_file(): + shutil.copy2(stub, dest) + if os.name != "nt": + dest.chmod(0o755) + return shim + + def control_agreement_replays_bytes(sample: Path, tmp: Path) -> None: """D4.1(a): on agreement the external result is the REFERENCE's — its raw - bytes, on both streams. - - Agreement is manufactured deliberately: the candidate is handed the exact - bytes the Python reference produces for this input, so the two engines - genuinely agree. That is the only way to reach this branch on Windows, - where a real candidate diverges from the reference on CRLF alone — and the - branch has to be right for when it becomes reachable, not merely for as - long as it is rare. + bytes, on BOTH streams. + + Two fixtures, because one of them could not carry the claim. + + (a) A real `python -m ownlang ownir` reference, with the candidate handed + the exact bytes it produced, so the two engines genuinely agree. This + is the case that proves the replay is byte-faithful against the actual + reference implementation. + + (b) A SYNTHETIC agreement in which both engines are the stub and the + fixture's stderr is deliberately non-empty. + + (b) exists because (a) alone was not load-bearing and was reported as + though it were. The assertion on stderr was guarded by `if + ref.stderr`, and a healthy reference writes none: CI printed `184 out, + 0 err` on Windows and `160 out, 0 err` on Linux, so the stderr half of + "both streams" was never compared on either platform while the control + said it had been. A guard that switches an assertion off exactly when + the data is ordinary is not a guard, it is a hole with a denominator + of zero — and this project does not get to spend a review cycle on + honest denominators and then ship one. + + The fixture bytes are chosen to survive nothing: CRLF that a text pipeline + would rewrite, and multi-byte UTF-8 that a decode-and-re-encode would + normalise. stdout is compared exactly; stderr is compared as a SUFFIX, + because the extraction step ahead of the compare sends dotnet's build + chatter to stderr and that chatter is not the launcher's replay. The + suffix is still the whole fixture, byte for byte. """ check = "ps1-agreement-replays" stub = stub_exe(tmp) @@ -313,7 +426,10 @@ def control_agreement_replays_bytes(sample: Path, tmp: Path) -> None: skip(check, "no dotnet/python") return - # Extract once, then ask the reference what it says about those facts. + problems = [] + reported = [] + + # --- (a) the real reference ------------------------------------------- facts = tmp / "agree.facts.json" ex = subprocess.run( [bash_exe(), str(ROOT / "scripts/own-check.sh"), @@ -329,37 +445,65 @@ def control_agreement_replays_bytes(sample: Path, tmp: Path) -> None: capture_output=True, check=False, cwd=str(ROOT), env={**os.environ, "PYTHONPATH": str(ROOT)}) - env = {"OWEN_RUST_CORE": stub, - "STAGE1_STUB_EXIT": str(ref.returncode)} out_file = tmp / "ref.out.bin" err_file = tmp / "ref.err.bin" out_file.write_bytes(ref.stdout) err_file.write_bytes(ref.stderr) - env["STAGE1_STUB_STDOUT_FILE"] = str(out_file) - env["STAGE1_STUB_STDERR_FILE"] = str(err_file) - - r = run_ps1(["-Engine", "compare", "-Format", "human", str(sample)], env=env) + r = run_ps1(["-Engine", "compare", "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": stub, + "STAGE1_STUB_EXIT": str(ref.returncode), + "STAGE1_STUB_STDOUT_FILE": str(out_file), + "STAGE1_STUB_STDERR_FILE": str(err_file)}) if r is None: skip(check, "no pwsh") return if r.returncode not in (0, 1): - skip(check, f"the engines did not agree, so the replay branch was not reached " - f"(exit {r.returncode}) [{tail(r)}]") + skip(check, f"the engines did not agree over the real reference, so the replay branch " + f"was not reached (exit {r.returncode}) [{tail(r)}]") return - - problems = [] if r.stdout != ref.stdout: - problems.append(f"stdout replay is not byte-faithful: {len(r.stdout)} bytes replayed vs " - f"{len(ref.stdout)} from the reference") - if ref.stderr and r.stderr != ref.stderr: - problems.append(f"stderr replay is not byte-faithful: {len(r.stderr)} bytes replayed vs " - f"{len(ref.stderr)} from the reference (a dropped stderr reads as a " - "silent run)") + problems.append(f"real reference: stdout replay is not byte-faithful — {len(r.stdout)} " + f"bytes replayed vs {len(ref.stdout)} from the reference") + reported.append(f"real reference {len(ref.stdout)} out") + + # --- (b) the synthetic agreement, with stderr that is actually there --- + syn_out = b"OWN001 synthetic finding\r\n\xe2\x80\x94 em dash, CRLF above\n" + syn_err = b"own-cli: warning: synthetic diagnostic\r\n\xc2\xa0nbsp then LF\n" + syn_out_file = tmp / "syn.out.bin" + syn_err_file = tmp / "syn.err.bin" + syn_out_file.write_bytes(syn_out) + syn_err_file.write_bytes(syn_err) + shim = _stub_as_python(tmp, stub) + r2 = run_ps1( + ["-Engine", "compare", "-Format", "human", str(sample)], + env={"OWEN_RUST_CORE": stub, + "PATH": f"{shim}{os.pathsep}{os.environ.get('PATH', '')}", + "STAGE1_STUB_EXIT": "0", + "STAGE1_STUB_STDOUT_FILE": str(syn_out_file), + "STAGE1_STUB_STDERR_FILE": str(syn_err_file)}) + if r2 is None: + skip(check, "no pwsh") + return + if r2.returncode not in (0, 1): + problems.append(f"synthetic: the engines did not agree (exit {r2.returncode}) — both " + f"sides are the same stub over the same bytes, so this is the launcher's " + f"comparison, not a real disagreement [{tail(r2)}]") + else: + if r2.stdout != syn_out: + problems.append(f"synthetic: stdout replay is not byte-faithful — {len(r2.stdout)} " + f"bytes replayed vs {len(syn_out)} in the fixture") + if not r2.stderr.endswith(syn_err): + problems.append( + f"synthetic: stderr replay is not byte-faithful — the reference wrote " + f"{len(syn_err)} bytes and they are not the tail of the {len(r2.stderr)} bytes " + f"the launcher emitted (a dropped stderr reads as a silent run)") + reported.append(f"synthetic {len(syn_out)} out, {len(syn_err)} err") + if problems: fail(check, "; ".join(problems)) else: - ok(check, f"agreement replays the reference's raw bytes on both streams " - f"({len(ref.stdout)} out, {len(ref.stderr)} err)") + ok(check, "agreement replays the reference's raw bytes on both streams " + f"({'; '.join(reported)})") def control_failure_evidence_exists(sample: Path, tmp: Path) -> None: From 157b6894961bbee64b32646c5d55d86834c3d578 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 08:37:27 +0000 Subject: [PATCH 22/22] evidence(stage1): both campaigns re-recorded, and "both streams" now has two graves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit p022-stage1-1 ubuntu, local 21/21 caught, 0 survived, 0 compile-error, 0 invalid, 0 runner-error p022-stage1-windows windows-latest 9/9 caught, same Both at source commit da897b42fcd7 with dirty:false, every expected catcher hit, and M00 surviving the unmutated tree on each. Nothing is added across those rows: they are different measurements on different platforms, and a mutant evaluated where its contract cannot be posed proves nothing about the surface it edits. The Windows record is the PUSH run's, not the pull_request run's. Both were 9/9, but the PR-triggered job checks out GitHub's synthetic merge commit and recorded `106973f375ce` — a commit that exists in no branch. Provenance that names a tree nobody can check out is not provenance, whatever its counts say. The two catchers this round exists for, from the Windows log: P03 -> ps1::ps1-agreement-replays stdout replay, stderr left intact P08 -> ps1::ps1-agreement-replays stderr replay alone, stdout intact Same catcher, different graves, and that is the whole point: measured separately before either was recorded, P03 dies on stdout in both fixtures with the stderr assertion silent, and P08 dies only on stderr against a launcher that emitted nothing there. Before this round one mutation broke both streams at once and the stdout assertion killed it on its own, so "both streams" was a sentence rather than a measurement. P09 -> ps1::ps1-absolute-locator the locator preflight keeps its POSITION P09 leaves every previously asserted observable correct — exit 2, the absolute requirement named, no verdict — and is caught only by the extraction counter this round added to the PowerShell surface. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb --- docs/evidence/p022-stage1-1.result.json | 48 +++++++++---------- docs/evidence/p022-stage1-windows.result.json | 38 ++++++++++----- docs/generated/p022-stage1-mutations.md | 15 ++---- 3 files changed, 56 insertions(+), 45 deletions(-) diff --git a/docs/evidence/p022-stage1-1.result.json b/docs/evidence/p022-stage1-1.result.json index b2af0616..9d528c1b 100644 --- a/docs/evidence/p022-stage1-1.result.json +++ b/docs/evidence/p022-stage1-1.result.json @@ -4,9 +4,9 @@ "campaign": "p022-stage1-1", "definition": "docs/evidence/p022-stage1-1.json", "definition_sha256": "c8bd2180f285a130dfadc153e966e48b2c2d8d6072d90651906112719c0040eb", - "source_commit": "1d649e99d9c9a8fac0913ac59fcbda94499479bb", + "source_commit": "da897b42fcd76ea1914d286bd6bae6065c072c70", "dirty": false, - "recorded_at": "2026-09-09T07:11:44Z", + "recorded_at": "2026-09-09T08:36:00Z", "layers": [ "stage1" ], @@ -15,7 +15,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 27.8 + "elapsed_seconds": 34.4 }, "mutations": [ { @@ -24,7 +24,7 @@ "catchers": [ "stage1::default-stays-python" ], - "elapsed_seconds": 24.1 + "elapsed_seconds": 27.6 }, { "id": "M02", @@ -32,7 +32,7 @@ "catchers": [ "stage1::rust-actually-runs-rust" ], - "elapsed_seconds": 22.5 + "elapsed_seconds": 25.5 }, { "id": "M03", @@ -41,7 +41,7 @@ "stage1::rc70-is-not-a-verdict", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 23.5 + "elapsed_seconds": 27.3 }, { "id": "M04", @@ -52,7 +52,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 23.6 + "elapsed_seconds": 27.1 }, { "id": "M05", @@ -60,7 +60,7 @@ "catchers": [ "stage1::raw-rc-retained" ], - "elapsed_seconds": 23.5 + "elapsed_seconds": 27.7 }, { "id": "M06", @@ -69,7 +69,7 @@ "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 23.8 + "elapsed_seconds": 26.9 }, { "id": "M07", @@ -78,7 +78,7 @@ "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 23.5 + "elapsed_seconds": 27.1 }, { "id": "M08", @@ -86,7 +86,7 @@ "catchers": [ "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 23.4 + "elapsed_seconds": 26.7 }, { "id": "M09", @@ -95,7 +95,7 @@ "stage1::absolute-locator-only", "stage1::bad-locator-is-2" ], - "elapsed_seconds": 23.3 + "elapsed_seconds": 27.4 }, { "id": "M10", @@ -105,7 +105,7 @@ "stage1::compare-same-input", "stage1::divergence-is-5" ], - "elapsed_seconds": 23.4 + "elapsed_seconds": 25.8 }, { "id": "M11", @@ -114,7 +114,7 @@ "stage1::compare-failure-classified", "stage1::exec-failure-is-5" ], - "elapsed_seconds": 22.9 + "elapsed_seconds": 26.2 }, { "id": "M12", @@ -122,7 +122,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 22.9 + "elapsed_seconds": 26.9 }, { "id": "M13", @@ -130,7 +130,7 @@ "catchers": [ "stage1::candidate-identity" ], - "elapsed_seconds": 22.8 + "elapsed_seconds": 26.7 }, { "id": "M14", @@ -139,7 +139,7 @@ "stage1::compare-no-substitution", "stage1::divergence-is-5" ], - "elapsed_seconds": 22.6 + "elapsed_seconds": 25.9 }, { "id": "M15", @@ -147,7 +147,7 @@ "catchers": [ "stage1::divergence-is-5" ], - "elapsed_seconds": 23.6 + "elapsed_seconds": 26.7 }, { "id": "M16", @@ -155,7 +155,7 @@ "catchers": [ "stage1::compare-zero-document" ], - "elapsed_seconds": 23.6 + "elapsed_seconds": 26.1 }, { "id": "M17", @@ -164,7 +164,7 @@ "stage1::absolute-locator-only", "stage1::locator-shapes" ], - "elapsed_seconds": 25.5 + "elapsed_seconds": 28.7 }, { "id": "M18", @@ -173,7 +173,7 @@ "stage1::absolute-locator-only", "stage1::locator-shapes" ], - "elapsed_seconds": 28.4 + "elapsed_seconds": 31.0 }, { "id": "M19", @@ -181,7 +181,7 @@ "catchers": [ "stage1::compare-failure-classified" ], - "elapsed_seconds": 23.3 + "elapsed_seconds": 26.5 }, { "id": "M20", @@ -195,7 +195,7 @@ "stage1::locator-shapes", "stage1::rust-failure-no-fallback" ], - "elapsed_seconds": 11.5 + "elapsed_seconds": 13.0 }, { "id": "M21", @@ -215,7 +215,7 @@ "stage1::rust-failure-no-fallback", "stage1::unexpected-rc-maps-to-5" ], - "elapsed_seconds": 15.5 + "elapsed_seconds": 17.2 } ] } diff --git a/docs/evidence/p022-stage1-windows.result.json b/docs/evidence/p022-stage1-windows.result.json index 718b78f8..d0fa7154 100644 --- a/docs/evidence/p022-stage1-windows.result.json +++ b/docs/evidence/p022-stage1-windows.result.json @@ -3,10 +3,10 @@ "comment": "Recorded mutation-campaign run (scripts/mutate_campaign.py --run). Raw facts only: outcomes, catchers, provenance. Counts are derived by scripts/render_checkpoint_status.py; regenerate this file by re-running the campaign, never by hand.", "campaign": "p022-stage1-windows", "definition": "docs/evidence/p022-stage1-windows.json", - "definition_sha256": "79ad1476f219239ecd4753d08dd8fd3ca018e17f35da368ba464196d1428aca0", - "source_commit": "1d649e99d9c9a8fac0913ac59fcbda94499479bb", + "definition_sha256": "6dbfa6054f6447bb41886b30b1b1ade3dcb9e1be0fb874dc5795b7775c30ebc3", + "source_commit": "da897b42fcd76ea1914d286bd6bae6065c072c70", "dirty": false, - "recorded_at": "2026-09-09T07:09:15Z", + "recorded_at": "2026-09-09T08:33:54Z", "layers": [ "ps1", "shapes" @@ -16,7 +16,7 @@ "id": "M00", "outcome": "survived", "catchers": [], - "elapsed_seconds": 66.6 + "elapsed_seconds": 58.9 }, "mutations": [ { @@ -25,7 +25,7 @@ "catchers": [ "ps1::ps1-absolute-locator" ], - "elapsed_seconds": 34.4 + "elapsed_seconds": 36.4 }, { "id": "P02", @@ -33,7 +33,7 @@ "catchers": [ "ps1::ps1-not-started-is-2" ], - "elapsed_seconds": 26.4 + "elapsed_seconds": 28.8 }, { "id": "P03", @@ -41,7 +41,7 @@ "catchers": [ "ps1::ps1-agreement-replays" ], - "elapsed_seconds": 24.1 + "elapsed_seconds": 31.6 }, { "id": "P04", @@ -49,7 +49,7 @@ "catchers": [ "ps1::ps1-failure-evidence" ], - "elapsed_seconds": 24.9 + "elapsed_seconds": 32.7 }, { "id": "P05", @@ -57,7 +57,7 @@ "catchers": [ "shapes::locator-shapes" ], - "elapsed_seconds": 24.2 + "elapsed_seconds": 31.1 }, { "id": "P06", @@ -68,7 +68,7 @@ "ps1::ps1-failure-evidence", "ps1::ps1-not-started-is-2" ], - "elapsed_seconds": 17.0 + "elapsed_seconds": 18.5 }, { "id": "P07", @@ -76,7 +76,23 @@ "catchers": [ "ps1::ps1-not-started-is-2" ], - "elapsed_seconds": 22.6 + "elapsed_seconds": 29.5 + }, + { + "id": "P08", + "outcome": "caught", + "catchers": [ + "ps1::ps1-agreement-replays" + ], + "elapsed_seconds": 28.8 + }, + { + "id": "P09", + "outcome": "caught", + "catchers": [ + "ps1::ps1-absolute-locator" + ], + "elapsed_seconds": 58.2 } ] } diff --git a/docs/generated/p022-stage1-mutations.md b/docs/generated/p022-stage1-mutations.md index f9ec029c..a6a87f02 100644 --- a/docs/generated/p022-stage1-mutations.md +++ b/docs/generated/p022-stage1-mutations.md @@ -12,7 +12,7 @@ Definition: `docs/evidence/p022-stage1-1.json` (sha256 `c8bd2180f285a130…`, 21 | measure | value | |--------------------------------------------------|---| -| recorded at commit | `1d649e99d9c9a8fac0913ac59fcbda94499479bb` | +| recorded at commit | `da897b42fcd76ea1914d286bd6bae6065c072c70` | | layers run (every one, for every mutation) | `stage1` | | mutations | 21 | | caught | 21 | @@ -55,10 +55,10 @@ Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `6dbfa6054f6447bb | measure | value | |--------------------------------------------------|---| -| recorded at commit | `1d649e99d9c9a8fac0913ac59fcbda94499479bb` | +| recorded at commit | `da897b42fcd76ea1914d286bd6bae6065c072c70` | | layers run (every one, for every mutation) | `ps1`, `shapes` | | mutations | 9 | -| caught | 7 | +| caught | 9 | | survived | 0 | | compile-error (no evidence either way) | 0 | | invalid-mutation | 0 | @@ -66,11 +66,6 @@ Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `6dbfa6054f6447bb | caught without every expected catcher | none | | honesty control `M00` (unmutated tree must pass) | survived — as required | -**This run is not evidence:** - -- the recorded result was taken over a different campaign definition (sha256 or campaign name differs) — re-run the campaign -- result/definition mutation sets differ (missing ['P08', 'P09'], unknown []) - | id | rule | mutation | outcome | caught by | |---|---|---|---|---| | P01 | ps1-locator-must-be-absolute | own-check.ps1 accepts a relative OWEN_RUST_CORE — Test-Path says the file is there, which is true and not the question D3 asks | caught | `ps1::ps1-absolute-locator` | @@ -80,5 +75,5 @@ Definition: `docs/evidence/p022-stage1-windows.json` (sha256 `6dbfa6054f6447bb | P05 | absolute-locator-is-accepted | the shell's drive-rooted arm stops matching — this is the defect that actually shipped: `[/\]` escapes the closing bracket, so the set is unterminated and matches NEITHER `C:/` nor `C:\`, and every correct Windows locator was refused as 'not absolute' while every Linux control stayed green | caught | `shapes::locator-shapes` | | P06 | absolute-locator-is-accepted | own-check.ps1's absoluteness test is inverted — the over-rejection direction on this surface: every fully qualified locator is refused as 'not absolute' and every relative one is admitted, which no assertion that only feeds it a relative path can see | caught | `ps1::ps1-absolute-locator`
`ps1::ps1-agreement-replays`
`ps1::ps1-failure-evidence`
`ps1::ps1-not-started-is-2` | | P07 | the-candidate-is-spawned-not-opened | own-check.ps1 goes back to invoking the candidate with the call operator — 'PowerShell runs it either way, why the ceremony?'. It does not run it: it asks the platform to OPEN it, so a file the loader cannot start is handed to a desktop handler (notepad on Windows, xdg-open on Linux), the run exits 0 with empty streams, and Owen reports a clean finding-free analysis of nothing | caught | `ps1::ps1-not-started-is-2` | -| P08 | ps1-agreement-replays-raw-bytes | own-check.ps1 drops the STDERR half of the agreement replay while leaving stdout byte-faithful — 'stderr is diagnostics, the result is stdout', which is how the original defect was written in the first place. It is the twin of P03 and exists because a single mutation that broke both streams could be killed by the stdout assertion alone: with this one, the stderr assertion is the only thing standing between the mutant and a green run | **not recorded** | — | -| P09 | the-locator-is-a-preflight | own-check.ps1 checks that the .NET toolchain answers before it validates the locator — 'fail on the missing tool first', which plenty of CLIs do. The rejection it then produces is still correct in every observable way: exit 2, the absolute requirement named, no verdict. What it loses is the POSITION: a decision reachable from one environment variable now happens behind an external process, and on the real path that process is a Roslyn extraction over the caller's whole tree. Nothing in this control caught that until it began counting child processes | **not recorded** | — | +| P08 | ps1-agreement-replays-raw-bytes | own-check.ps1 drops the STDERR half of the agreement replay while leaving stdout byte-faithful — 'stderr is diagnostics, the result is stdout', which is how the original defect was written in the first place. It is the twin of P03 and exists because a single mutation that broke both streams could be killed by the stdout assertion alone: with this one, the stderr assertion is the only thing standing between the mutant and a green run | caught | `ps1::ps1-agreement-replays` | +| P09 | the-locator-is-a-preflight | own-check.ps1 checks that the .NET toolchain answers before it validates the locator — 'fail on the missing tool first', which plenty of CLIs do. The rejection it then produces is still correct in every observable way: exit 2, the absolute requirement named, no verdict. What it loses is the POSITION: a decision reachable from one environment variable now happens behind an external process, and on the real path that process is a Roslyn extraction over the caller's whole tree. Nothing in this control caught that until it began counting child processes | caught | `ps1::ps1-absolute-locator` |