Skip to content

feat: assert the cross-layer invariants with verify.sh - #347

Merged
chubes4 merged 1 commit into
mainfrom
verify-invariants
Aug 7, 2026
Merged

feat: assert the cross-layer invariants with verify.sh#347
chubes4 merged 1 commit into
mainfrom
verify-invariants

Conversation

@chubes4

@chubes4 chubes4 commented Aug 6, 2026

Copy link
Copy Markdown
Member

The pattern

Every defect that reached a live site during this rollout was a disagreement between layers, not a fault within one:

seam what it cost
unit User= vs unit HOME= the agent would have started with a home it cannot read
recorded option vs manifest capture read a set the site had moved on from
manifest vs harvest components a hardcoded list drifted from the declaration
owned set vs permission.edit declared editable, actually denied
function vs the lib its caller sources a declare -F guard silently skipped, leaving database credentials group-writable

Each component was internally correct and individually tested. Nothing owned the space between them, so nothing failed until someone looked.

The looking is what doesn't scale. An operator with two sites inspects a rendered systemd unit before starting services. An operator with two thousand does not.

What it does

./verify.sh          # exits non-zero on any disagreement
./verify.sh --json   # machine-readable, for a scheduled check

Three seams: owned-set agreement (option ↔ manifest ↔ permissions, plus the denies that keep the agent off a payment gateway), service-identity coherence (User=HOME=, and any value still pointing into a home the install migrated away from), and owned-mode completeness.

Two design choices

It reports and never repairs. A check that fixes what it finds can't be trusted to report honestly, and the failure it hides is the one worth seeing. upgrade.sh is the repair; this is the measurement.

An invariant it cannot evaluate is skip, not pass. My own test caught me violating this: the manifest-writability check needs root to become www-data, and unprivileged it reported a failure rather than declining to answer — which would have cried wolf on every CI run and taught everyone to ignore it. That's how a checker becomes decoration.

Validated against live installs

h44lacrosse.com: 15 passed, 0 failed, 0 skipped. chubes.net (workspace mode): owned-set checks correctly skip.

Then deliberately re-broken, one seam at a time — it named every one:

FAIL  kimaki.service: User=opencode but HOME=/root — the service cannot read its own home
FAIL  kimaki.service: 'KIMAKI_DATA_DIR=/root/.kimaki' points outside opencode's home
FAIL  manifest disagrees with the recorded set — capture and permissions would diverge
FAIL  manifest missing — out-of-band capture cannot learn the editable set
FAIL  NOT denied: wp-content/plugins/** — third-party code and core are writable

The first two are the exact bug hand-fixed on h44 this morning. Live config was restored and re-verified clean afterwards.

Tests assert it catches each of those, not merely that it passes when healthy — a checker green on both a healthy and a broken install is worse than none, because it converts "nobody looked" into "something looked and said it was fine".

Every defect that reached a live site during the managed-hosting rollout
was a disagreement BETWEEN layers, not a fault within one:

  unit User= vs unit HOME           the agent ran with a home it cannot read
  recorded option vs manifest       capture read a set the site had moved on from
  manifest vs harvest components    a hardcoded list drifted from the declaration
  owned set vs permission.edit      declared editable, actually denied
  function vs the lib its caller sources
                                    a declare -F guard silently skipped, leaving
                                    database credentials group-writable

Each component was internally correct and individually tested. Nothing owned
the space between them, so nothing failed until somebody looked — and the
looking is what does not scale. An operator with two sites inspects a
rendered systemd unit before starting services. An operator with two
thousand does not.

verify.sh owns the seams. It exits non-zero on disagreement and is cheap
enough to run on a schedule.

It reports and never repairs. A check that fixes what it finds cannot be
trusted to report honestly, and the failure it hides is exactly the one
worth seeing; silent convergence is how "it works on my box" survives.
upgrade.sh is the repair, this is the measurement.

An invariant it cannot evaluate is reported as SKIPPED, not passed. My own
test caught me violating that: the manifest-writability check needs root to
become www-data, and on an unprivileged run it reported a failure instead of
declining to answer — which would have cried wolf on every CI run and taught
everyone to ignore it.

Validated against both live installs. h44lacrosse.com: 15 passed, 0 failed.
chubes.net (workspace mode): the owned-set checks correctly skip. Then
re-broken deliberately, one seam at a time, and it named each one —
including the exact User=opencode/HOME=/root mismatch that was hand-fixed
this morning.
@chubes4
chubes4 merged commit f7c47a6 into main Aug 7, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant