Skip to content

Stabilize nightly mutation targets and fault injection - #31

Open
jeregrine wants to merge 1 commit into
mainfrom
fix-ci-campaign-regressions
Open

jeregrine wants to merge 1 commit into
mainfrom
fix-ci-campaign-regressions

Conversation

@jeregrine

Copy link
Copy Markdown
Member

Summary

Fix the three failures in the latest nightly October 11 and October 12 campaigns:

  • Resolve mutation targets by file and literal test/property name instead of hard-coded line numbers. Missing or ambiguous names still fail closed, and the local PR gate now validates every target and production replacement before nightly qualification.
  • Force test-only lease expiry in one shard system turn. Previously a heartbeat could renew the backdated lease, or another timer could invalidate the queued tick, before expiry ran.
  • Wait for cluster controls to arrive in the suspended receiver's mailbox before asserting their exact revision order. A completed RPC to the sender does not establish delivery to another node.

No production code, timeouts, or fault-coverage budgets change.

Evidence

  • Before: qualification aborted at the stale distributed_test.exs:5842 selector; the same failure pattern appears in September runs. After: all 57 unique baseline targets pass and all 72 compiling protocol mutants are killed.
  • The queued-heartbeat regression fails with the old expiry helper and passes with atomic expiry. It and the existing repeated-expiry regression pass 31 repetitions with one peer scheduler; all three focused regressions pass 31 repetitions with four peer schedulers.
  • Selector fixtures confirm that moved tests still run, adjacent tests are excluded within the module, and missing/ambiguous names are rejected before baseline execution.

The full multi-hour, six-profile Jepsen soak has not been rerun; this does not claim to resolve historical soak failures on older commits.

Merge Danger

Blast radius: test selection and fault-injection synchronization only. The library API, wire protocol, and production lease behavior are unchanged; reverting restores the previous test harness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant