Mutation testing: the harness, seventeen vectors, the triage record - #64
Merged
Conversation
tools/mutate.py breaks the reference in one small way at a time and asks whether the vector corpus notices. It generates every mutant of every module in python/bitlisp from a fixed edit set (comparison operators flipped to their neighbor or negation, arithmetic and bitwise operators swapped, and/or exchanged, integer constants moved by one, booleans flipped, if tests negated, not removed, raise deleted, break/continue exchanged), runs the corpus against each mutant in a private mirror of the tree, and reports the survivors with their diffs. With --tests, survivors also run the pytest suite, separating what nothing catches from what the tests catch and the corpus does not. Mirrors copy python/ and tools/ and link vectors/ and puzzles/, so mutants never touch the checkout and parallel workers never see each other's edits. The unmutated tree must pass every oracle the mutants face before any mutant runs, otherwise a broken mirror would report every mutant killed (which is how the first pass's --tests column went wrong before puzzles/ was linked). Timeouts count as kills. The unit test checks that every module yields parseable, distinct mutants, that a constant mutant changes exactly its constant, that the mirror shares data and copies code, and that a broken error-code table dies in the corpus. CLAUDE.md gains the command.
Nineteen cases, each a behavior the spec states that no vector exercised, found by tools/mutate.py as mutants the corpus let survive. Every case passes the reference. The two path cases were cross-checked against the consensus oracle (chia-rs, flags 0), the three seal cases against the vendored Bitcoin Core framework. VM.md section 3.1: an interior zero byte in a path atom is not a leading zero byte and is not costed (path_interior_zero_byte, path_leading_and_interior_zero_bytes). The surviving mutant counted every zero byte. VM.md section 2 (D5): the one-byte atom 0x7f in the long form is non-minimal (nonminimal_one_byte_atom_7f), a lone 0xfc prefix byte is bad_encoding (lone_prefix_fc), and a length of 8,191 in the three-byte form is non-minimal (nonminimal_length_e0_at_8191). The corpus pinned the 0x40 boundary but not the 0x2000 one. CONDITIONS.md time asserts: ASSERT_SEQUENCE_HEIGHT accepts 0 and rejects -1 (seqheight_zero, seqheight_negative). The other three time asserts had both cases, this one had neither. CONDITIONS.md section 1, VALIDATION.md rule 6: a reserved declared cost of exactly -1 is bad_condition_arg, not reserved_cost_too_low (reserved_cost_minus_one). The corpus had -500. CONDITIONS.md self asserts: ASSERT_MY_SCRIPTPUBKEY and ASSERT_MY_AMOUNT take exactly one operand (four arity cases). Only the outpoint and taproot asserts had arity cases. CONDITIONS.md message family: a pair in a non-amount specifier field is bad_condition_arg (assure_script_specifier_pair), an empty ASSERT_ANNOUNCEMENT is bad_condition_arity (assert_announcement_arity_zero), and a specifier composing amount with tapleaf parses its fields in operand order (assure_amount_tapleaf_specifier_parses). No composed case had carried an amount before an identity field. VALIDATION.md rule 3 (C9): an amount specifier over a zero-amount prevout balances (amount_specifier_over_zero_amount_input_balances). The amount domain is 0 to MAX_MONEY and the corpus started at 1. CONDITIONS.md seals: the compact-size boundary at 253 in the txid and outputs hash, a 252-byte and a 253-byte output script under SEAL_OUTPUTS and 253 outputs under SEAL. The one-byte form was the only one any seal vector exercised.
docs/mutation-triage.md records what tools/mutate.py found on its first pass and how each survivor was judged: the method, the six survivor classes (equivalent, unreachable guard, same code, model precondition, beyond reach, gap) with the reason each of the first five is accepted without a vector, the per-module numbers on main at PR 60 (1,733 mutants, 1,551 killed, 179 corpus survivors of which the pytest suite kills 55, 3 timeouts), the nineteen gaps with the mutant that exposed each and the vector that closes it, and the representative sites of every accepted survivor. Line numbers are left out on purpose: the classes are what a later pass compares against, and the re-running section says how. docs/README.md gains the entry.
EvanWinget
force-pushed
the
mutation-testing
branch
from
August 24, 2026 04:22
00ac043 to
c3fa730
Compare
A mutant that makes the reference raise outside its error taxonomy stops the corpus runner on an escaping exception before any vector reaches a verdict. That is detection, but by Python rather than by the corpus, and counting it as a kill overstates what the corpus guards. The corpus driver now exits with its own code on a vector's verdict, so any other nonzero exit is classified crashed, and the pytest pass treats only exit 1 (failing tests) as a kill. The summary gains the column and the JSON report carries the verdict. The unit test pins the mapping.
Six findings from the multi-agent review of PR 64, each reproduced before fixing. The runner's malformed-case catch turned ValueError and KeyError escaping a mutated reference into vector verdicts, inflating kill counts. run_vectors gains MalformedCase, a VectorError subclass raised on that path, and the corpus driver classifies it as a crash. The driver stopped at the first failing file in alphabetical order, so a mutant's killed-versus-crashed verdict was an artifact of file ordering and every VM kill first paid the condition and validation files. It now continues past crashing files, lets a vector verdict win over a crash, and visits files cheapest first, in an order the baseline run measures. Under --tests the pytest oracle collected the mirror's own copy of the harness test, which mutated the already-mutated package, built nested mirrors, and manufactured false kills. That test and the corpus-runner test (a repeat of the run every survivor just passed) are excluded, and a generated conftest disables Hypothesis deadlines so parallel suites on a loaded machine cannot fail a latency check and report a false kill. The report keeps each run's output tail so a test kill names its failing test. Negating an if test whose condition is a bare equality-class comparison or a "not" duplicated the comparison swap or the "not" removal, one semantic mutant under two ids. Those negation sites are skipped, and AugAssign, IfExp, and While now get the operator swaps and test negations the docstring already claimed, closing the untouched augmented-assignment and expression-test sites the review counted, the message ledger's "+= 1" among them. Worker mirrors are removed at exit instead of accumulating in the temporary directory. Crashed and timed-out mutants are listed by site like survivors. __init__ is excluded from the inventory: its mutants would measure the export list, not the reference.
The two 253-byte seal cases are removed. The review ran the mutation both ways: the pre-existing seal boundary case and the seal unit suite already pin the 253 side of the compact-size boundary, and with both cases deleted every _compact_size mutant is still killed, so only the 252 case, which is kept, closes a gap. The largest case in the corpus pinned nothing. The two path-cost cases are renamed from interior to trailing zero bytes: programs 0100 and 000100 place the zero byte at the tail, and the old names claimed a position the bytes do not have. Case content is unchanged.
The full re-run after the review fold-ins counts 1,651 mutants: 1,241 killed by the corpus, 225 crashed (detected by Python before any verdict, no kill credit), 180 survived, 5 timed out. The pass table gains the crashed column, and the numbers paragraph is rewritten from the new run: the pytest suite kills 52 of the 180 survivors and 128 survive both oracles. Three former test kills were artifacts of the mirror's own harness test mutating the already-mutated package and now survive genuinely: the error-code guard in errors.py, a frozen flag in conditions.py, and sha256tree's argument index in operators.py. Each already sits in an accepted class. The one new survivor, from the newly mutated expression tests, is the taproot-versus-scriptpubkey name choice in validation.py's unsatisfied-scriptpubkey message. It only names a condition in a message and joins the same-code class beside the name table index.
Owner
Author
Review pass (/code-review 64 max, Fable finders, adversarial verify)Ten confirmed findings. Six are folded into 19a725e, two are vector corrections in dac86ba, and two were dropped at the finding cap but resolved anyway, outcomes below.
Final pass after the fold-ins: 1,651 mutants, 1,241 killed by the corpus, 225 crashed, 180 survived (52 killed by the pytest suite, 128 survive both, all 180 in the accepted classes), 5 timeouts. Corpus at 1,149 cases with this PR's 17 vectors. Lint clean, CI green. |
The suite runner aborted a vector file at the first case whose exception escaped the error taxonomy, so a mutant that a later case in the same file properly killed was reported as crashed whenever the escaping case sorted earlier. Reproduced on the machine.py mutant that deletes the operator-not-atom raise: dispatch.json's fourth case raised AttributeError and ended the file, while its forty-fourth case failed with a proper vector verdict under the same mutant. Swapping the two cases flipped the verdict. The runner now runs every case in the file. A vector verdict on any case is the file's verdict. Only a file with no verdict at all raises MalformedCase, for the first case that escaped, with the escaping exception chained as its cause so the standalone runner can print its traceback under the FAIL line. The catch widens from (KeyError, ValueError) to Exception. The MalformedCase docstring already defined it as an implementation raising something no vector can expect, and the narrower tuple let an IndexError from a mutated deserializer traceback out of the standalone runner with no FAIL line and no summary. The one place that draws the crash-versus-verdict line is now the suite runner, and the mutation harness's driver no longer needs a bare except of its own to agree with it.
Findings from the second multi-agent review of PR 64, each reproduced before fixing. The baseline run read the vector file order out of the last 2,000 characters of the timing run's output, the same tail the report keeps. The 38 current files fill 1,477 of those characters, and once the lines outgrew the tail (thirteen more files, or one warning on stderr) the earliest files would silently drop out of every mutant's run with no error anywhere. The driver now appends timings to a file the baseline names through the environment, and the baseline reads that file whole. The corpus driver was a 25-line program inside an f-string, unchecked by ruff, and the baseline hand-built the same argv that run_corpus built. The driver is now the corpus_driver function in this module, run in the mirror through a one-line bootstrap that imports the mirror's copy, and the baseline calls run_corpus like every worker. The baseline collapsed killed, crashed, and timeout into None and discarded the output, so a slow machine or a broken venv was reported as the unmutated tree failing its oracle. baseline_order now raises BaselineError naming which oracle failed with what verdict and the output tail. A misspelled --module value filtered the inventory to nothing, ran the whole baseline, printed an all-zero table, and exited 0. Unknown module names are refused with exit 2 before anything runs, and --module and --only are mutually exclusive instead of --only silently discarding --module. The mirror synthesized a python/tests/conftest.py to load a no-deadline hypothesis profile, which would have clobbered any real conftest the suite later gained. The profile is now registered by a checked-in conftest and selected with --hypothesis-profile=mutate, so the harness runs exactly the files CI runs. Cleanups the review verified: the by-name __init__ exclusion excluded nothing (that module yields no mutation site) and its comment described mutants that cannot exist. The __pycache__ unlink loop ran over a directory the mirror never has, since the copy skips it and every subprocess runs with PYTHONDONTWRITEBYTECODE. The "not" removal joins the site visitor through the same in-place rewrite the raise deletion uses, retiring the separate transformer and the "not"-suffixed ids. The per-mutant result is a Result dataclass instead of a positional 4-tuple. The unused site ordinal is gone, the original module's unparse is cached across diffs, and the --timeout help says that the suite and the baseline get SUITE_TIMEOUT_FACTOR times the budget.
The triage record's table is re-run against main at PR 66 merged with this branch, on the harness after the second review pass. The rule that a case's verdict outranks an earlier case's escape moved about forty crashes into the kill column (185 crashes against 225), and the taproot assert's removal on main took 38 mutants out of the inventory. Survivors are 180 as before, all in the accepted classes, with one more pytest kill (the frozen flag of the derived-taproot output) and one fewer validation survivor (the removed assert's message fallback). The crash rule in the method section is restated at case granularity, the lone-0xFC gap row carries the same crashed-not-killed caveat the ASSERT_ANNOUNCEMENT row does, and the first pass's numbers are kept as history under the counting they were made with.
EvanWinget
changed the title
Owner
Author
Second review pass (/code-review 64 high, Fable finders, adversarial verify)Fourteen confirmed findings, on the branch after the first pass's fold-ins. Outcomes:
Smaller items also folded in: Re-run on 2026-09-05 against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Eleven commits, in review order. The first three are the work, the rest are review fold-ins and the merge of
main.tools: the mutation harness.tools/mutate.pygenerates small semantic mutants of every module inpython/bitlisp/(flipped comparisons, swapped operators, off-by-one constants, negatediftests, deletedraise, exchangedbreak/continue, removednot) and runs the vector corpus against each in a private mirror of the tree. Survivors are reported with diffs.--testsruns the pytest suite over corpus survivors.python/tests/test_mutate.pycovers the generator, the mirror, and one end-to-end kill.CLAUDE.mdgains the command,ci/lint/codespell-ignore-words.txtgainsNotIn(theastclass).vectors: the gaps the first mutation pass found. Cases across eight files, each a behavior the spec states that no vector exercised. The commit message cites the spec section for each.vectors/README.mdgains a mutation-coverage section.docs: the mutation triage record.docs/mutation-triage.mdwith the method, the survivor classes and why each is accepted, the pass numbers, the gap table, and the representative accepted survivors.docs/README.mdentry.tools: crashes counted apart from kills,tools: review fold-ins for the mutation harness,vectors: two review corrections,docs: the final pass numbers with crashes counted apart. The first review pass (comment below): escaping exceptions classified as crashes rather than kills, the driver continuing past crashing files, the nested-mirror false kills under--tests, the duplicate negation sites, the two 253-byte seal cases removed as pinning nothing.mainat PR 66.tools: a case's verdict outranks an earlier case's escape. The suite runner ran a file to its first escaping exception, so a mutant killed by a later case in the same file was counted as crashed. Every case now runs, and a verdict anywhere wins.tools: review fold-ins for the mutation harness, second pass. The baseline's file order no longer travels through a 2,000-character output tail, the corpus driver is a checked function rather than an f-string program, baseline failures name the oracle and its output, an unknown--moduleis refused with exit 2 instead of exiting 0 on an empty inventory, the hypothesis profile is a checked-in conftest selected by flag, and the dead__init__exclusion and__pycache__loop are gone.docs: the pass of 2026-09-05. The table re-run on the merged tree with the final harness.No spec change. No implementation change in
python/bitlisp/: every gap was in the corpus, not the reference.Numbers
On
mainat PR 66 merged with this branch, corpus at 1,126 cases: 1,613 mutants, 1,243 killed by the corpus, 185 crashed, 180 survived, 5 timeouts. The pytest suite kills 53 of the survivors, 127 survive both. Every survivor is in one of five accepted classes: equivalent mutants, guards no input reaches, checks that share an error code with the one that then fires, the transaction model's own preconditions intx.py, and boundaries no vector can carry (the 2^20 and 2^27 length forms, a 12 GB atom, a signature scalar above the group order, a 65,536-byte script).What the gaps were
Seventeen vectors. The ones worth a reviewer's attention:
vm/paths.json). Cross-checked against chia-rs.0x7fin the long form, a lone0xfcprefix, and a length of 8,191 in the three-byte form were all unpinned non-minimal encodings (vm/serialize.json). The corpus pinned the 0x40 boundary and not the 0x2000 one.ASSERT_SEQUENCE_HEIGHThad no zero and no negative case where its three siblings had both.reserved_cost_too_low.ASSERT_MY_SCRIPTPUBKEYandASSERT_MY_AMOUNThad no arity cases.validation/seals.json, cross-checked against the vendored Core framework).Review guide
docs/mutation-triage.mdfirst: it is the argument for what was and was not turned into a vector..venv/bin/python tools/mutate.py --only tx:0 --listprints a diff for any id from the report. Ids are ordinal within a module and shift when the module changes, which is why the record names sites rather than ids..venv/bin/python tools/run_vectors.py(1,126 cases),.venv/bin/pytest python/tests,ci/lint/lint.sh. The full mutation pass is.venv/bin/python tools/mutate.py --tests, about 40 minutes on 8 workers.--module costsruns one module in a couple of minutes.A note on the harness itself
The first
--testscolumn reported every survivor killed. The mirror lackedpuzzles/, which two test modules read at collection, so pytest failed before running a test. The harness now linkspuzzles/and refuses to start unless the unmutated mirror passes the same oracles the mutants face, and says which oracle failed when it refuses.