feat: assert the cross-layer invariants with verify.sh - #347
Merged
Conversation
Every defect that reached a live site during the managed-hosting rollout
was a disagreement BETWEEN layers, not a fault within one:
unit User= vs unit HOME the agent ran with a home it cannot read
recorded option vs manifest capture read a set the site had moved on from
manifest vs harvest components a hardcoded list drifted from the declaration
owned set vs permission.edit declared editable, actually denied
function vs the lib its caller sources
a declare -F guard silently skipped, leaving
database credentials group-writable
Each component was internally correct and individually tested. Nothing owned
the space between them, so nothing failed until somebody looked — and the
looking is what does not scale. An operator with two sites inspects a
rendered systemd unit before starting services. An operator with two
thousand does not.
verify.sh owns the seams. It exits non-zero on disagreement and is cheap
enough to run on a schedule.
It reports and never repairs. A check that fixes what it finds cannot be
trusted to report honestly, and the failure it hides is exactly the one
worth seeing; silent convergence is how "it works on my box" survives.
upgrade.sh is the repair, this is the measurement.
An invariant it cannot evaluate is reported as SKIPPED, not passed. My own
test caught me violating that: the manifest-writability check needs root to
become www-data, and on an unprivileged run it reported a failure instead of
declining to answer — which would have cried wolf on every CI run and taught
everyone to ignore it.
Validated against both live installs. h44lacrosse.com: 15 passed, 0 failed.
chubes.net (workspace mode): the owned-set checks correctly skip. Then
re-broken deliberately, one seam at a time, and it named each one —
including the exact User=opencode/HOME=/root mismatch that was hand-fixed
this morning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pattern
Every defect that reached a live site during this rollout was a disagreement between layers, not a fault within one:
User=vs unitHOME=permission.editdeclare -Fguard silently skipped, leaving database credentials group-writableEach component was internally correct and individually tested. Nothing owned the space between them, so nothing failed until someone looked.
The looking is what doesn't scale. An operator with two sites inspects a rendered systemd unit before starting services. An operator with two thousand does not.
What it does
Three seams: owned-set agreement (option ↔ manifest ↔ permissions, plus the denies that keep the agent off a payment gateway), service-identity coherence (
User=↔HOME=, and any value still pointing into a home the install migrated away from), and owned-mode completeness.Two design choices
It reports and never repairs. A check that fixes what it finds can't be trusted to report honestly, and the failure it hides is the one worth seeing.
upgrade.shis the repair; this is the measurement.An invariant it cannot evaluate is
skip, notpass. My own test caught me violating this: the manifest-writability check needs root to becomewww-data, and unprivileged it reported a failure rather than declining to answer — which would have cried wolf on every CI run and taught everyone to ignore it. That's how a checker becomes decoration.Validated against live installs
h44lacrosse.com: 15 passed, 0 failed, 0 skipped. chubes.net (workspace mode): owned-set checks correctly skip.
Then deliberately re-broken, one seam at a time — it named every one:
The first two are the exact bug hand-fixed on h44 this morning. Live config was restored and re-verified clean afterwards.
Tests assert it catches each of those, not merely that it passes when healthy — a checker green on both a healthy and a broken install is worse than none, because it converts "nobody looked" into "something looked and said it was fine".