Skip to content

Guard every already-covered file with a repo-wide coverage ratchet#4385

Open
TaprootFreak wants to merge 11 commits into
developfrom
feat/coverage-gate-expansion
Open

Guard every already-covered file with a repo-wide coverage ratchet#4385
TaprootFreak wants to merge 11 commits into
developfrom
feat/coverage-gate-expansion

Conversation

@TaprootFreak

@TaprootFreak TaprootFreak commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Why

The repo has exactly one coverage gate today: jest.frick.config.js, covering 7 files — 0.4 % of production code. Everywhere else, a change can drop coverage to zero and CI stays green.

What this does

Measures the whole suite once and pins every production file that already reaches 100 % on all four metrics. A drop below 100 on any pinned file now fails CI.

before after
Pinned files 7 399
Share of production code 0.4 % 25.1 %

This freezes coverage that already exists — no test was added to raise it.

One existing test is rewritten (see below), which means the pre-existing checks and sharded test jobs run a changed test too. This PR is therefore not configuration-only.

What it deliberately does not claim

It is a regression gate, not a quality statement. Overall coverage is 57.66 % of statements and 39.46 % of branches — the pinned set is simply what is complete today.

Of the 399 files, 217 carry real logic (functions and/or branches) and 182 are purely declarative (NestJS modules, constant files). They are kept as two commented blocks in the config so the number is not mistaken for test depth. Pinning the declarative ones is intentional: the moment logic is added to one, it must arrive with tests.

Measurement

Two properties matter, and docs/coverage-gate.md records both so the list stays reproducible:

  1. Full compilation (tsconfig.coverage.json, isolatedModules: false) — same as the Frick gate. Transpile-only emit reports phantom uncovered branches on injected constructors and would understate dozens of files.
  2. Whole suite runs — files are frequently covered by specs other than their own.

The gate job runs without the Postgres service the sharded test job uses, matching the measurement. The suites it enables are not in the pinned list, and running them can only raise coverage.

The Frick gate stays separate and untouched

It runs only its seven specs and therefore proves those specs alone reach 100 % — an assertion a whole-suite run cannot make, since there any spec may contribute the coverage. Two gates, two different questions.

Verification

  • Gate passes with no threshold violation: 299/306 suites, 5170 tests, and it passed in CI as well (~16 min)
  • Parallel execution yields byte-identical coverage totals to a --runInBand reference run — hence no --runInBand in the CI script
  • Both failure modes verified against jest 29.7 rather than assumed: a file dropping below 100 exits 1 with threshold ... not met, and a deleted or renamed pinned file exits 1 with Coverage data ... was not found. The gate cannot silently stop protecting a file.
  • format:check green, workflow YAML parses, config loads with 399 keys

Inventory

The collection glob matches 1,643 files under src/; 1,591 carry instrumentable code and appear in the report. The remaining 52 are type-only files that compile to no executable statements and therefore cannot be pinned. Test scaffolding (shared/utils/test.util.ts, shared/utils/test.shared.module.ts — 60 and 28 importers, all specs) is excluded, so an untested change to a test helper cannot fail a production gate.

One test changed, and why it belongs here

bitcoin-fee.service.spec.ts asserted parallel execution by wall-clock time (duration < 100ms for five mocked 10 ms calls). This gate runs the whole suite a second time per PR, which doubles the exposure to that timing assertion — a flaky required check is the worst kind, because it teaches a team to ignore red.

Checking it turned up more than flakiness. Replacing Promise.all in getTxFeeRates with a sequential loop left the old test green (27 passed): five sequential 10 ms calls total ~50 ms, under the limit. The assertion could only fail from machine load, never from the defect it existed to catch — it failed at 181 ms under load in exactly that way.

The test now counts how many calls are in flight at once and requires that maximum to equal the txid count. Against the same sequential mutation it fails with Expected: 5, Received: 1. Stable and, for the first time, effective. Coverage of the pinned BitcoinBasedFeeService is unchanged at 100 %.

Scope boundary

1062 partially covered and 130 uncovered files stay open — closing them means writing tests, which is deliberately not part of this change. docs/coverage-gate.md lists the gaps by area and names the 29 files already at ≥90 % on all four metrics as the next batch, so the boundary is documented rather than silent.

The only coverage gate so far was the Frick one: seven files, 0.4% of production
code. Nothing stopped a change from silently dropping coverage anywhere else.

Measure the whole suite once and pin every production file that already reaches
100% on branches, functions, lines and statements - 401 of 1593 files. Dropping
below 100 on any of them now fails CI.

This is a regression gate, not a claim about test quality. Overall coverage is
57.7% of statements and 39.5% of branches; the pinned set is simply what is
complete today. Of the 401, 218 carry real logic and 183 are declarative
(modules, constants) - kept as separate blocks in the config so the count is not
mistaken for test depth. The declarative ones are pinned on purpose: adding
logic to one of them now requires tests.

Two properties of the measurement matter and are documented so the list stays
reproducible:

- Full compilation via tsconfig.coverage.json, as the Frick gate already does.
  Transpile-only emit reports phantom uncovered branches on injected
  constructors, which would understate dozens of files.
- The whole suite runs, because files are frequently covered by specs other than
  their own.

The Frick gate stays untouched and separate: running only its seven specs, it
proves those specs alone reach 100%, which a whole-suite run cannot assert.

Verified locally: the gate passes with no threshold violation (299/306 suites,
5170 tests), and parallel execution yields byte-identical coverage totals to a
--runInBand reference run, which is why the CI script does not serialise.

docs/coverage-gate.md records the measured gaps, including the 29 files already
at >=90% on all four metrics as the next batch.
Two files under shared/utils are test support, not production code:
test.util.ts (60 importers) and test.shared.module.ts (28), every one of them
a *.spec.ts. Both were collected and pinned, which made two things wrong.

It coupled a production gate to test scaffolding: extending a test helper
without testing the helper itself would have turned CI red although no
production coverage moved. And it made the documented inventory inaccurate,
since both were counted as production files.

Exclude them from collectCoverageFrom and drop their threshold entries.
Pinned files 401 -> 399 (217 with logic, 182 declarative).

Correct the remaining numbers against a fresh verified run: 1,591 measured
files, statements 57.66%, branches 39.46%, functions 31.84%. The collection
glob matches 1,643 files; the 52 that never appear in the report are type-only
files that compile to no executable statements, so they cannot be pinned.

Also document what happens when a pinned file is deleted or renamed - jest
exits 1 with "Coverage data ... was not found", so the gate cannot silently
stop protecting a file - and fix the stale "three job groups" comment in the
workflow, which this change turns into four.
Two documentation defects, both about claims a reader cannot check.

The inventory said all 52 uncollected files are type-only. Two of them are not:
p2b.service.ts and payin/enums/index.ts consist entirely of commented-out code.
The counts and the conclusion were right, but lumping dead code in with type
declarations hides a second, different reason a file cannot be measured. Both
are pre-existing and left untouched; removing them is separate cleanup.

The parallelism claim rested on a one-off measurement ("a --runInBand run
produced identical totals"), which nothing in the repo lets a maintainer
reproduce. Replace it with the structural reason it holds: istanbul merges
per-worker counters additively, so scheduling cannot turn a covered statement
into an uncovered one. The measurement stays as corroboration, not as the
argument. Likewise drop the precise "~16 min" from the workflow comment and
date the approximation in the doc, so neither goes stale silently.
Dating a measurement records when someone asserted it, not whether it holds.
The doc still carried two claims a maintainer cannot check from the repository:
that a serial reference run once produced identical totals, and that this job
is the slowest of the four at roughly a quarter hour.

Drop both. What remains is derivable from the repository itself: coverage
merging is additive across workers, so parallelism cannot lower a per-file
percentage, and the gate runs the whole suite under full compilation while the
test job shards three ways and the Frick gate runs seven specs. That explains
the runtime without asserting a number nobody can reproduce.

Measured values stay in the pull request, where they belong - a PR reports what
its author ran, with the CI run alongside it. Durable documentation is read
when nobody remembers who measured what.
The test named "should execute in parallel" measured wall-clock duration and
required it to stay under 100 ms while five mocked calls each slept 10 ms. That
assertion answered the wrong question in both directions.

It was too strict under load: scheduling delay alone pushed it to 181 ms on a
busy machine, failing a test whose subject was correct.

It was also too weak to catch what it existed for. Sequential execution of five
10 ms calls totals about 50 ms, comfortably under the limit. Verified by
replacing Promise.all in BitcoinBasedFeeService.getTxFeeRates with a sequential
loop: the old test stayed green, 27 passed. It could only ever fail because of
machine speed, never because of the defect it was written to detect.

Assert the property itself instead. The mock now counts how many calls are in
flight simultaneously and the test requires that maximum to equal the number of
txids. Concurrent execution reaches five; a sequential implementation never
exceeds one. Against the same sequential mutation the new test fails with
"Expected: 5, Received: 1", so it detects the regression the old one missed,
and it does not depend on timing at all.

Coverage is unchanged: the pinned BitcoinBasedFeeService still reports 100% on
all four metrics under the full gate run.
…e claim

CONTRIBUTING requires the `// --- NAME --- //` form for major sections; the two
principal blocks of the gate config used a different decorative style.

The doc also claimed the pinned list "was derived from a run without" Postgres.
That is a statement about a past measurement, and nothing checked in records it.
State the current, checkable behaviour instead: no pinned file belongs to the
migration suites MIGRATION_TEST_PG enables, and enabling further suites can only
raise coverage.
The same unverifiable statement removed from docs/coverage-gate.md survived in
the workflow: "matching the measurement the pinned list was derived from".
Nothing checked in records that historical environment, and the workflow and
the documentation should not make different-strength claims about the same
fact. State only what a reader can check: no pinned file belongs to the
migration suites that need Postgres.
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

6 review passes to zero findings, two independent lenses each round (conformance against CONTRIBUTING, and logic/correctness), re-run in full against origin/develop...HEAD after every fix rather than as a delta.

What the rounds turned up:

Round Findings
1 Test scaffolding (test.util.ts, test.shared.module.ts — 60 and 28 importers, all specs) was pinned as production; stale "three job groups" comment
2 Inventory misclassified 2 of 52 uncollected files; unverifiable measurement claims
3 Dating a claim does not make it checkable — removed what the repo cannot confirm
4 The gate exposed a pre-existing flaky test; section comments off-convention; a provenance claim
5 Scope description understated that an existing test changes; same provenance claim surviving in the workflow
6 none, both lenses

The round-4 finding was the substantive one. The should execute in parallel test asserted concurrency via wall-clock time. Replacing Promise.all in getTxFeeRates with a sequential loop left it green — five sequential 10 ms calls total ~50 ms, under its 100 ms limit. It could only fail from machine load, which it did at 181 ms, never from the defect it was written to catch. It now counts simultaneously in-flight calls; against the same mutation it fails with Expected: 5, Received: 1.

Mechanism verified rather than assumed, against jest 29.7's CoverageReporter:

  • A pinned file dropping below 100 exits 1 with threshold ... not met
  • A pinned file deleted, renamed or excluded exits 1 with Coverage data ... was not found — the gate cannot silently stop protecting a file
  • Threshold keys resolve via path.resolve from the working directory; both the npm script and the workflow run from the repo root
  • The 182 declarative pins are not vacuous: adding an unexecuted function moves a metric from 0/0 to 0/N and fails

All 12 checks green, including the new ratchet job at ~15 min.

The threshold block spelled out the same four-metric object for each of the 399
pinned files, which made the config 2037 lines and turned every rename into a
five-line diff. The paths now live in PINNED_LOGIC and PINNED_DECLARATIVE and
coverageThreshold is built from them, so the two groups are data rather than a
comment, and adding a file is one line.

The pinned set itself is unchanged: same 399 paths, same order, same grouping.

Also emit an lcov report, which the workflow uploads when the gate fails.
The job had no timeout and would have held a runner for the default six hours
if it hung. It is also the most expensive job in the workflow: an unsharded
whole-suite run under full compilation.

When the gate fails, jest names the file and the metric but not the uncovered
lines, which left a local rerun of roughly fifteen minutes as the only way to
find out what to fix. The lcov report is now uploaded as an artifact instead.

Also run the gate with --silent, matching the test script. An unsilenced
whole-suite run buries the threshold message under the console output of every
suite.
The assertion sits between the point where the mocked getMempoolEntry calls are
parked and the point where they are released. On failure the release never ran,
leaving five pending promises and a getTxFeeRates call that could not settle.
Moving the release into a finally block keeps the failure clean.
Three gaps a reader would otherwise find the hard way:

- A newly added production file with no coverage passes the gate. The list only
  protects what is already on it, and it grows by hand.
- When a purely declarative file first gains executable logic, the function
  metric moves from 0/0 to 0/N and the gate turns red. Tests stay the preferred
  fix, but unpinning that one file is an allowed outcome when the pull request
  says so. For the logic-carrying files the rule is unchanged.
- The lcov report is uploaded as an artifact when the gate fails.

Also describe the two pinned-path arrays, since that is where a file is added
now.
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Four review findings addressed, plus three follow-ups split out as issues.

Fixed here

The 2037-line threshold block The 399 paths now live in PINNED_LOGIC and PINNED_DECLARATIVE, and coverageThreshold is generated from them. 2037 → 458 lines, one line per file instead of five, and the two groups are data rather than a comment. The pinned set is byte-identical: same 399 paths, same order, same grouping.
A red gate was not diagnosable jest names the file and the metric but not the uncovered lines, so the only way to find out what to fix was a ~15-minute local rerun. The config now emits lcov and the job uploads coverage-gate/ as an artifact on failure (7-day retention).
The job had no timeout timeout-minutes: 30. It is the most expensive job in the workflow, and without a bound a hang would have held a runner for the default six hours.
A pending promise survived a failing assertion In bitcoin-fee.service.spec.ts the assertion sits between parking the mocked calls and releasing them, so on failure the release never ran. The release moved into a finally.

docs/coverage-gate.md also gained the three things it was silent about: that a newly added file with no coverage passes the gate, that unpinning a purely declarative file which first gains logic is an allowed outcome when the PR says so, and where the lcov artifact comes from.

Verification

  • Whole-file check that the pinned set survived the refactor: 399 paths, identical in both membership and order, all four metrics still at 100, collectCoverageFrom and the header comment untouched.
  • bitcoin-fee.service.spec.ts: 27/27 green.
  • The rewritten parallelism test still catches the defect it exists for — replacing Promise.all in getTxFeeRates with a sequential loop fails it with Expected: 5, Received: 1, and with the finally in place the run now ends cleanly instead of leaving the call unsettled.
  • tsc --noEmit clean, prettier --check clean on src/**/*.ts and on the gate config, workflow YAML parses with the expected job structure.

Follow-ups, tracked separately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant