feat(stories): delivery-record checks — the record against itself, and against GitHub

[ophiocus]

TL;DR — Read-only checks that the delivery record agrees with itself (the turn_git_evidence row against its two story-evidence events, at the start and at settlement) and with GitHub (the head SHAs the mirror stored as a witness). One module that returns findings; it surfaces nothing yet, gates nothing, and needs no migration. 27 tests, each "break the promise → the report names it". Where findings should show up is a product choice, proposed separately so it can be discussed on its own.

Follow-on to #264, taking @javiertoledo's "up to you" as the brief. #289 retired the receipt machinery and, with it, the only verifier the tree had — deliberately, and I think rightly: the mediation layer was where the hard defects lived (#195, #233, #226 were all bugs in the machinery of distrust, not in the work it guarded). What 0.12 kept instead is a delivery record that Insights, the story timeline, and turn attribution all stand on — and nothing can notice when that record stops agreeing with itself. This PR adds the noticing, as a read, using only what 0.12 already writes.

Credit where it starts: @Julian-Genuario's #226, @javiertoledo's re-verification and diagnosis on #264, and @adrian-lorenzo's #289, whose double-write is the reason this needs no digest of its own.

The class, in one sentence

Forward-only verification can't see deletion. — #264

A verifier that enumerates from the ledger it is verifying can only confirm what survived; absence has no row to iterate. The old fix was a reverse pass in one file. That file is gone, but the new tree writes each turn's facts in more than one place from more than one code path, so the reverse pass becomes cross-ledger reads — every check enumerates from a ledger other than the one it verifies, and reports over that denominator:

check	enumerates from	verifies
completeness	turns (settled)	turn_git_evidence
coherence	the evidence row	the turn:{id}:context and turn:{id}:git story-evidence events
attribution	GitHub's reported head SHAs	both internal tellings

What 0.12 already writes (nothing new is written to detect)

1. Two tellings of every turn, at both ends. TurnGitEvidenceService.start() writes the row's initial branch/SHA and a turn:{id}:context event with the same starting point; complete() writes the final SHA, commits, changed files and dirty state into turn_git_evidence and a turn:{id}:git event with the same facts — or, on the failure path, a captureError on the row and a git.capture_failed event under the same key. Two ledgers, one writer; the comparer was the only thing missing.
2. A third telling on a channel the database does not write. The mirror stores the head SHAs GitHub reports — branches, pull heads, CI heads — arriving over the HMAC-verified webhook and the authenticated mirror scan. The mirror already uses finalSha == headSha to link a pull request to the exact turn that produced it. Read the other way, those heads are a witness: content-addressed, and not ours to rewrite.

The checks — each one a promise 0.12 makes, made checkable

Completeness — "Final SHA, commits, changed files, and dirty state are captured when it settles, including failure and cancellation paths." Every settled turn carries either a completed capture or an explicit captureError. A settled turn with neither is missing-evidence or unsettled-capture — except one the dispatcher claimed and then failed or canceled before TurnGitEvidenceService.start() ran (budget rejection, credential issue, environment preparation, cancel while queued). Such a turn has neither telling by construction; it is counted as unstartedCaptures and is not a finding. A succeeded turn cannot have skipped start(), and a context event that survives without its row means the row was destroyed, so both still flag.

Coherence — the row and its events agree. At the start: the context event exists (missing-context) and matches the row on initialSha, initialBranch, workspaceId, engineSessionId. At settlement: a completed capture is paired with git.changes_recorded and agrees on initialSha, finalSha, the commit sequence and dirty; a capture error is paired with git.capture_failed. A missing pair is missing-event; a wrong or disagreeing one is divergent. Insights and the timeline read from this substrate, so divergence means the surfaces are wrong, not merely inconsistent.

Attribution — "GitHub facts are linked to an exact Facility turn when their head SHA matches that turn's recorded final SHA." Presence of a SHA somewhere in GitHub's record is not enough — an old head is still a real SHA — so a telling is corroborated only when three things hold: GitHub reported its final SHA on the turn's own branch (taken from the context event, a telling the row did not write); the SHA is not the turn's starting point, unless the telling records no commits (an honest no-op); and the telling's own commit list ends at that SHA (the writer logs --reverse). A turn whose row is corroborated is witnessed (witnessedTurns makes coverage visible). When the two internal tellings disagree about the final SHA and exactly one side is corroborated, the finding becomes witness-disagrees and names the side that drifted. Both corroborated, or neither: it stays a coherence finding — the witness never guesses. An unwitnessed turn is not a finding; it simply hasn't travelled that far yet.

Proof — every shape is "break the promise, watch the report"

services/api/test/delivery-record.integration.test.ts (16 tests, scoped fixture org):

mutate	the report says
clean lifecycle (row, context event, git event all agree)	0 findings
honest capture error paired with its git.capture_failed event	0 findings
delete the evidence row of a settled turn	completeness: missing-evidence
null finalSha and erase captureError	completeness: unsettled-capture
delete the turn:{id}:context event	coherence: missing-context
context event names a different starting SHA	coherence: divergent — "row and context event disagree on initialSha"
capture error on the row, its git.capture_failed event never landed	coherence: missing-event
completed capture, but the event on record says the capture failed	coherence: divergent
delete the turn:{id}:git event	coherence: missing-event
change the event's finalSha	coherence: divergent
…then GitHub reports the row's SHA	attribution: witness-disagrees — "the evidence event drifted"
GitHub reports the event's SHA instead	"the evidence row drifted"
GitHub has seen neither	stays coherence: divergent
walk the record with limit: 4 and the cursor	same findings, same witnessedTurns, nothing double-counted
another org's project	never inspected

Review round — @adrian-lorenzo's three findings

services/api/test/delivery-record-review.integration.test.ts (11 tests). Each fixture is modelled on the writer that produces it — the dispatcher claims a turn (running, startedAt set) before the budget check and only reaches evidence.start() after credentials and environment preparation; complete() logs commits --reverse; the mirror stores branch heads by name, pull heads by head ref, and one CI observation per GitHub event.

finding	now	proof
a normal budget rejection was missing-evidence	neither telling exists ⇒ unstartedCaptures, no finding	budget-failed turn (claimed, startedAt set) and a turn canceled while queued: 0 findings, counted 2; a failed turn whose context event survives but whose row is gone, and a succeeded turn with neither: still missing-evidence
an old GitHub SHA could make the checker blame the correct event	verdicts need exactly one corroborated side: on the turn's branch, not its starting point, ending its own commit list	the turn's own initial SHA written into finalSha: no conviction; a head seen on an unrelated branch: no conviction; the honest case on the branch: still convicts; an honest no-op turn: still witnessed
limit: 1 loaded every CI observation for a matching SHA	witnessObservations returns distinct (sha, branch) pairs server-side, CI joined to its pull for the head ref; exported so the bound is observable	1,001 CI events for one head → 1 pair; 5 SHAs asked → the 4 pairs the mirror holds

One fixture in the original suite changed: the "row drifted" case witnessed the event's SHA on an unrelated branch — the shape the review rejected. The witness now arrives as a pull opened from the story branch and the event's commits end at that SHA; the assertion is unchanged.

Both suites: 27/27, three consecutive runs; tsc and biome clean on the rebased branch.

Bounded by construction

Read-only; keyset-paginated over turns on the existing (orgId, projectId) scope; every companion query — evidence rows, events, and the GitHub witness lookup — is an IN over the page, and the witness lookup returns distinct pairs, never one row per observation. Memory does not grow with org history (#295 absorbed rather than inherited). Findings are counted against checkedTurns — all settled turns — never against survivors.

Limits, stated rather than papered over

• The witness corroborates SHAs, and only for work that reached GitHub. Commit lists and dirty state have no external telling in 0.12; they are checked ledger-to-ledger only. A corruption that rewrites a telling's final SHA and its commit list consistently would still be corroborated; timestamps are deliberately not used, because branch and pull rows bump updatedAt on every sync and CI observations can be backfilled.
• A failed turn that lost both tellings is indistinguishable from one that never began capture. The turn event stream would be the next ledger to consult; out of scope here.
• The settled turn is the unit. A turn deleted whole is visible only from outside the unit — a GitHub head no turn claims, which reconciliation already keeps as an honest, unattributed category.
• This observes accidental corruption, control-plane bugs, reconciliation drift and silent data loss. It is not a defence against a principal with write access to both internal ledgers; only the external witness constrains that, and only where it overlaps.

Non-goals — and what is deliberately not in this PR

No receipts, no approvals, no leases, no per-agent profiles, no gating of any operation, ever. No migration.

And no surface: this PR adds no attention items, changes no health, schedules no job. The checks are facts about the record; where those facts should appear — and whether a finding should ever change what a person sees on a story — is a product call I would rather discuss than assume. Those come as separate proposals.

Files

• services/api/src/stories/delivery-record.ts — verifyDeliveryRecord, witnessObservations (new)
• services/api/test/delivery-record.integration.test.ts
• services/api/test/delivery-record-review.integration.test.ts

stories/evidence.ts and turns/git-evidence.ts — the two writers — are untouched.

Verify locally

docker compose -f docker-compose.dev.yml up -d postgres
pnpm --filter @facility/api exec vitest run test/delivery-record.integration.test.ts test/delivery-record-review.integration.test.ts
pnpm --filter @facility/api test
pnpm typecheck && pnpm lint

🤖 Generated with Claude Code

[adrian-lorenzo]

Thanks for following this through on the new delivery record.

Please address three issues before merge: normal budget rejections are flagged as missing evidence even though capture never starts; an old GitHub SHA can make the checker blame the correct event when only the row was corrupted; and limit: 1 still loads every CI observation for a matching SHA—1,001 rows in my probe.

Please distinguish unstarted captures, avoid assigning blame from SHA presence alone, and bound the witness queries, with regression tests for each.

[ophiocus]

Thanks @adrian-lorenzo. All three addressed in fe5d305: unstarted captures are counted rather than flagged; a verdict requires one corroborated side (the SHA on the turn's branch, not its starting point, ending the telling's own commit list) instead of SHA presence; witness reads return distinct pairs. The "Review round" section in the body has the table and the 11 regression tests. The workspace-e2e and self-host-build failures are the moby source fetch in the runner-image build, which passed on main at the same base commit; a rerun should clear them, as I cannot rerun from the fork.