fix(db): P1 — root-membership backfill crashes backend+security when root org id diverges - #680
Merged
Merged
Conversation
…ID in backfill
fuzefront-backend and fuzefront-security have been crashlooping in prod for
~4h. Root cause: migration 022 (monolith) / 015 (security) hardcoded
ROOT_ORG_ID ('...0010') into the organization_memberships backfill INSERT.
On the 2026-07-29 prod rebuild, migration 014/015's own "adopt a
pre-existing platform org rather than create a second one" branch adopted
92f2020b-2bdb-41f0-98ff-1ef759b41741 (slug 'fuzefront') as the root org and
never created a '...0010' row. Every subsequent boot's INSERT then violated
organization_memberships_organization_id_foreign (23503) -- uncaught by
ON CONFLICT DO NOTHING, which only dedupes committed conflicts, not a failed
insert -- and knex propagated the error out of initializeDatabase(), taking
down both services on every restart.
Fix: resolve the root org the same way portalRepository.ensureRootPortal()
already does at runtime -- prefer the row whose id is ROOT_ORG_ID, else fall
back to the oldest organizations row of type='platform' -- and backfill
against ITS id. Never creates a second platform org (the 'fuzefront' slug
is already taken), so no dual-root reconciliation is introduced by this fix.
Skips outright (self-heals later) if no platform org exists at all yet.
Applied identically to both copies (022 monolith mirror + 015
security-service authoritative). Adds a regression test reproducing the
exact prod condition (ROOT_ORG_ID absent, only an adopted platform org
exists) plus a re-run idempotency check.
Because migration 022/015 failed on every prior prod boot, it was never
recorded in knex_migrations -- this fix is expected to let both services
recover on redeploy with NO manual prod DB write required.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session-Id: 00fca8e1-2252-4c20-b8c3-5d2baebb0a60
Contributor
Automated code review (gate-code-review)Credit balance is too low Report-only — this check never blocks merge. |
izzywdev
added a commit
that referenced
this pull request
Aug 26, 2026
…iberately replaced SECOND independent master regression, nine minutes after the first. Fixing the apps fixture (b462827) turned "Run apps routes tests" green and moved the failure to "Generate test coverage" -- which runs the FULL suite, so it was failing all along behind the earlier step. Both steps are red on master's latest run (32933352569); this is the other one. Bisected to 73c30aa (#680, 05:15Z today): "root-membership backfill crashes backend+security when root org id diverges". It changed migration 022 and left rootOrgAbsentGuards.test.ts asserting the pre-fix behaviour. The test named "skips the backfill when the root organization does not exist" supplied a fixture that DOES contain a platform org, just under a different id (legacy-platform-id). Post-#680 that org IS the root org: 022 resolves it the same way ensureRootPortal() does -- prefer ROOT_ORG_ID, else the oldest type='platform' row -- so it adopts and backfills. The fixture contradicted the test's own name. Inverting the assertion back would restore the 2026-08-16 P1 crashloop: with a prod DB whose platform-root org has an id other than ROOT_ORG_ID and no ROOT_ORG_ID row at all (the 2026-07-29 rebuild, 92f2020b-..., slug `fuzefront`), hardcoding ROOT_ORG_ID made every INSERT violate organization_memberships_organization_id_foreign on every boot. The migration is right; the guard was stale. So the assertion is updated to the intended contract, and pinned where it matters: expect(inserts[0].bindings).toContain('legacy-platform-id') expect(inserts[0].bindings).not.toContain(ROOT_ORG_ID) That second line IS the #680 regression, expressed as a test. NOT a weakened test -- coverage of the skip branch is added, not removed. The pre-existing empty-fixture test ({users: [], organizations: []}) cannot distinguish "skipped the backfill" from "had nobody to backfill", so a new case covers the real skip: users present, no platform org, zero membership inserts. Both branches are now genuinely exercised; before this, one was asserted backwards and the other only vacuously. Verified with CI's own command against a local PostgreSQL 16: npx jest --coverage --runInBand --testPathIgnorePatterns="permit-integration|billing-" before: 1 failed, 548 passed, 549 total after: 550 passed, 550 total (41 suites) rootOrgAbsentGuards alone: 7 passed, 7 total. With b462827 this should take Backend Tests green for the first time since 05:07Z, unblocking every PR in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv
izzywdev
added a commit
that referenced
this pull request
Aug 26, 2026
… PAT repos + census roster reconcile (#811) * fix(ci): fleet-pat-health watched 3 of the 9 repos holding GH_RELEASE_PAT The file's own header says "Add new fleet repos here when they receive GH_RELEASE_PAT". Nine repos hold it; FLEET_REPOS named three. The other six had NO expiry detection at all -- which is the single thing this workflow exists to provide, absent for two thirds of its subjects. FuzeCall, FuzeDeploy and FuzeMerchandize were provisioned the credential today via FuzeSDLC's provision-secrets.yml (run 32948657038, 3x SET GH_RELEASE_PAT), so they are added in the same change rather than left to be noticed later. FuzePlan already held it and was simply never listed. The list is not hand-assembled: it is exactly the repos whose OWN workflows reference secrets.GH_RELEASE_PAT to bump a prod tag, from the fleet-wide survey in provision_secrets.py. FuzeFront and FuzeSDLC are deliberately EXCLUDED despite holding the secret. Neither uses it to cut a release -- FuzeFront for this health check itself, FuzeSDLC for secret provisioning -- so listing them would query a release workflow that does not exist and report a failure that is not one. That hazard is real for a repo that IS in scope: FuzeMerchandize references the PAT from build-and-push.yml, not release.yml, and querying an absent workflow returns the same "Could not query" error as an expired credential. A false alarm here is not cosmetic -- it trains people to ignore the alert. So entries now accept an optional `=workflow.yml` override, defaulting to WORKFLOW_FILE. Verified: YAML parses; the entry parser resolves plain entries to release.yml and the override to build-and-push.yml. NOT labelled auto-merge. master is deploy-on-push in this repo, so merging is a production deploy -- merge in a deploy window, per the convention in #794. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * fix(census): reconcile the portal roster with the live registry — 4 defects it reported about itself Census run 32949523555 against production emitted four warnings that were about the ROSTER, not about production, and each one manufactured a spurious MISSING row. A MISSING that means "this file is wrong" is indistinguishable from one that means "a product dropped out of the portal", which defeats the only thing this roster exists to detect. Three entries carry a slug the registry has never served: keys -> fuzekeys fuzecontact -> contact fuzehub-ventures -> fuzehub All three were reported MISSING while simultaneously appearing under their real slug in the "in the registry but not in expected-portal-apps.json" warnings -- the same product counted twice, once as absent and once as unexpected. One product was absent from the roster entirely: `clock`, which the registry returns and builtins.ts seeds. Corrected in the direction _meta.notASlugMigrationWorklist mandates: "If a live registry slug genuinely differs from an entry below, FIX THIS FILE to match the registry -- never the other way around." No product's registered slug is touched, per CLAUDE.md's immutable-slug rule. FuzeQuality is REMOVED rather than corrected. PR #810 (merged 05:07 today) deleted it from builtins.ts and added migration 011_suspend_phantom_fuzequality_builtin, because the product has no repository. Notably the same census still found it `activated` in prod serving a 200-that-is- HTML, so #810's migration has NOT run there yet. With the entry gone that now surfaces as an "in the registry but not expected" warning -- which is the correct signal for a phantom that is still live, and better than the FAIL row it produced while pretending to be a real product. Net effect on the same production data: MISSING drops from 9 to 6, and those 6 are genuinely unregistered -- fuzex, fuzebi, deploy, call, fuzeplan, merchandize. No PASS or FAIL row changes; this corrects the roster, not the verdict. Verified: `node --test scripts/check-portal-federation-health.selftest.mjs` 5/5 pass (includes the three broken-input proofs and the anti-vacuity check). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * fix(test): create the BOLA fixture's "other" org — it was referenced but never inserted master has been red on Backend Tests since 05:07Z today. Bisected to d44ee55 (#809): green on the merge commit before it (02c2b44, #808), red on it and on every run since. This is the cause. #809 added the describe block "GET /api/apps - visibility/org scoping parity, and apps.organization_id NOT NULL". Its beforeAll declares otherOrgId = uuidv4() and then inserts an app with organization_id: otherOrgId -- but never inserts the organizations row. apps.organization_id carries the FK apps_organization_id_foreign, so that insert aborts the whole beforeAll and every test in the block fails. `otherOrgId` appears exactly three times in the file: declaration, assignment, and use as a foreign key. The block never passed. REPRODUCED, not inferred. Stood up the PostgreSQL 16 server already present in the image (no docker daemon in this container, so no compose/kind) and ran CI's own recipe -- workspace-root `npm ci`, `npm run db:init`, then `jest --testPathPattern=apps --runInBand`: before: Tests: 4 failed, 37 passed, 41 total all 4 -> "insert or update on table \"apps\" violates foreign key constraint \"apps_organization_id_foreign\"" after: Tests: 41 passed, 41 total Also ran the job's other pattern, tests/(auth|auth-oidc): 99 passed, 99 total -- so this block was the only failure in Backend Tests, and the fix does not mask a second one. The org is owned by ADMIN_USER_ID because organizations.owner_id is NOT NULL with an FK to users, and both seeded users (admin, demo) are the block's two callers, so neither can be a neutral third party. That does not weaken the BOLA assertion: scopeAppsQuery (src/routes/apps.ts:211) scopes on memberOrgIds ONLY -- membership, never ownership -- and no organization_memberships row is created for this org, so the app stays excluded for both callers exactly as intended. Verified against that function, not assumed. Cleanup deletes both orgs, and only after the apps referencing them are gone: apps.organization_id has no ON DELETE, so the reverse order fails. Rides on this branch because #811 is itself blocked by this failure and cannot go green until it lands; it is a master hotfix, unrelated to #811's own subject. SEPARATE, NOT FIXED HERE: #809 and #810 each added a migration numbered 011 in backend/applications/src/migrations (011_apps_organization_id_not_null and 011_suspend_phantom_fuzequality_builtin), an hour apart, neither able to see the other. Ordering between them is now an alphabetical tiebreak. Renaming a migration that may already have run against a live database is not a change to make blind, so it is flagged rather than folded in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * chore(gitignore): ignore backend/database.sqlite, written by npm run db:init Reproducing a CI failure locally (npm run db:init) writes backend/database.sqlite, and nothing ignored it -- so it showed up as an untracked file that a stop-hook or a careless `git add -A` would invite committing. A database binary must never be committed: it churns on every run and can carry real data. Ignored rather than committed, and the stray file removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * fix(test): migration 022's guard still asserted the contract #680 deliberately replaced SECOND independent master regression, nine minutes after the first. Fixing the apps fixture (b462827) turned "Run apps routes tests" green and moved the failure to "Generate test coverage" -- which runs the FULL suite, so it was failing all along behind the earlier step. Both steps are red on master's latest run (32933352569); this is the other one. Bisected to 73c30aa (#680, 05:15Z today): "root-membership backfill crashes backend+security when root org id diverges". It changed migration 022 and left rootOrgAbsentGuards.test.ts asserting the pre-fix behaviour. The test named "skips the backfill when the root organization does not exist" supplied a fixture that DOES contain a platform org, just under a different id (legacy-platform-id). Post-#680 that org IS the root org: 022 resolves it the same way ensureRootPortal() does -- prefer ROOT_ORG_ID, else the oldest type='platform' row -- so it adopts and backfills. The fixture contradicted the test's own name. Inverting the assertion back would restore the 2026-08-16 P1 crashloop: with a prod DB whose platform-root org has an id other than ROOT_ORG_ID and no ROOT_ORG_ID row at all (the 2026-07-29 rebuild, 92f2020b-..., slug `fuzefront`), hardcoding ROOT_ORG_ID made every INSERT violate organization_memberships_organization_id_foreign on every boot. The migration is right; the guard was stale. So the assertion is updated to the intended contract, and pinned where it matters: expect(inserts[0].bindings).toContain('legacy-platform-id') expect(inserts[0].bindings).not.toContain(ROOT_ORG_ID) That second line IS the #680 regression, expressed as a test. NOT a weakened test -- coverage of the skip branch is added, not removed. The pre-existing empty-fixture test ({users: [], organizations: []}) cannot distinguish "skipped the backfill" from "had nobody to backfill", so a new case covers the real skip: users present, no platform org, zero membership inserts. Both branches are now genuinely exercised; before this, one was asserted backwards and the other only vacuously. Verified with CI's own command against a local PostgreSQL 16: npx jest --coverage --runInBand --testPathIgnorePatterns="permit-integration|billing-" before: 1 failed, 548 passed, 549 total after: 550 passed, 550 total (41 suites) rootOrgAbsentGuards alone: 7 passed, 7 total. With b462827 this should take Backend Tests green for the first time since 05:07Z, unblocking every PR in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * chore(gitignore): ignore jest coverage output npm run test:coverage writes ~6.6M of generated HTML + lcov per workspace, and nothing ignored it — so reproducing backend-tests.yml's "Generate test coverage" step locally leaves a large untracked directory that a careless `git add -A` would sweep in. CI produces the same output and ships it to Codecov from the runner; it is never a source artifact. Unanchored (`coverage/`) so it matches any workspace, not just backend/. Second of the same class as backend/database.sqlite: running CI's own recipe locally generated artifacts this repo had no rule for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * fix(migration): applications-service 011 threw when there was nothing to fail about THIRD independent master regression from the same 05:06-05:15Z merge burst, and the only one that is a production hazard rather than a test-only one. Found by reproducing ci.yml's "Applications service (unit + integration)" job locally -- a DIFFERENT suite from the two already fixed on this branch, which live in backend/tests (the monolith). tests/migrations.idempotency.integration.test.ts ✕ is a clean no-op against a pre-existing apps table/enum/columns ✕ creates portal_apps idempotently alongside the pre-existing apps schema Error: organizations.00000000-0000-0000-0000-000000000010 (the platform root org) does not exist yet -- backend/src migration 015_seed_root_platform_organization must run before this one. #809 added this migration with an unconditional throw when the root org is absent. Its stated reasoning is sound and is preserved: backfilling to an absent id would violate apps_organization_id_foreign, and a clear error beats a mystery FK stack trace later. But the guard fired even when there was NOTHING TO BACKFILL. The root org is seeded by a DIFFERENT deployable's migration tree (backend/src's 015), so any environment where applications-service migrates first -- including a fresh/schema-only DB -- hits an exception. A migration that throws is a boot crashloop, not a warning. That is precisely the 2026-08-16 P1 shape that backend/src's migration 022 was fixed for, now reintroduced one tree over. The idempotency test is not incidental: it runs THIS tree, standalone, against a bare schema, and asserts a clean no-op. That contract is the thing #809 broke. Narrowed to the case that warrants it: - root org absent AND org-less rows exist -> still throws, now naming the count so the operator knows what is at stake - root org absent AND zero org-less rows -> logs and returns DEFAULT and NOT NULL are deliberately NOT set in the skip branch: the DEFAULT would point at a non-existent org and reintroduce the exact FK hazard the guard exists to prevent. Two claims I checked rather than assumed, because both would have been wrong: 1. "self-heals on the next boot" -- FALSE under knex, which records the migration as applied and never re-runs it. What actually closes the gap is the SIBLING migration on the same shared table, backend/src's 026, in the tree that also owns 015 and therefore always has the root org by then. That is its own header's stated contract: "whichever service's migrations happen to run first against a given database does the real work, and the other is a no-op". The comment says this, not the convenient version. 2. The query-side half still holds. service.ts's list() may drop its `organization_id IS NULL` arm only once no row can be null -- and the skip branch is taken precisely when there are ZERO org-less rows, so nothing can silently disappear. Verified locally against PostgreSQL 16 with ci.yml's own recipe (core build, applications-service type-check, then the suite): before: 2 failed, 204 passed, 206 total (1 of 14 suites red) after: 206 passed, 206 total (14 suites) tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv * fix(census): restore fuzequality to the roster — removing it was my error An earlier commit on this branch removed `fuzequality` from scripts/expected-portal-apps.json on the premise that it was not registered. That premise was wrong, and the live census disproves it. Census run 32960147607 (2026-08-26 10:49:51Z) against app.fuzefront.com: fuzequality FuzeQuality activated FAIL https://app.fuzefront.com/apps/ fuzequality/assets/remoteEntry.js — remoteEntry returned 200 but is HTML (content-type 'text/html') — SPA fallback answering a 404 with 200 It is `activated` in the registry, and it is one of the four entries in backend's BUILTIN_MANIFESTS (`fuzesocial`, `fuzeagent`, `clock`, `fuzequality`) — the only four whose slug comes from FuzeFront's own seed. It is a first-party product that belongs in the expected roster. The consequence of dropping it was not cosmetic. This file is the ONLY thing that makes a MISSING app detectable — an app absent from both the registry and this roster is invisible to the census entirely. Removing a real product converts it from a detectable failure into a blind spot, which is the exact failure mode the file exists to prevent (see its `_meta` header). Worth being precise about what the census result means for it: `fuzequality` is NOT healthy. It returns a 200 whose body is HTML — an SPA fallback answering a 404 — which the checker correctly scores as FAIL. Restoring the entry does not assert the product works; it asserts the product is expected, so that its brokenness keeps being reported instead of disappearing. Verified: `node --test scripts/check-portal-federation-health.selftest.mjs` → 5/5 pass against the edited roster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv --------- Co-authored-by: izzywdev <izzy.weinberg@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1 prod incident fix.
fuzefront-backendandfuzefront-securityhave been in CrashLoopBackOff (~4h+) in Contabo prod.Root cause
Migration
022_root_membership_backfill_and_personal_org_reclassify(and its authoritative twinbackend/security/src/migrations/015_...) backfilled root memberships with a hardcodedROOT_ORG_ID(…0010). Prod has no…0010row — the real platform root is92f2020b-…(adopted on the 2026-07-29 rebuild via migration 015's "adopt existing platform org, don't create …0010" branch). The INSERT hit FKorganization_memberships_organization_id_foreign(23503);ON CONFLICT DO NOTHINGdoes not catch an FK failure; knex propagated the throw out ofinitializeDatabase()→ crashloop on every boot.Fix
Resolve the actual root org the same way
ensureRootPortal()does — prefer theROOT_ORG_IDrow, else the oldesttype='platform'org; skip (self-heal) if none — and backfill against its id. No new org created (avoids a dual-root), no manual DB write. Applied identically to both copies. Regression test added for the "…0010 absent, adopted92f2020bpresent" case.Recovery
Migration 022 failed, so it is not recorded in
knex_migrations— knex re-runs it on next boot. Once this image deploys, the backfill resolves92f2020b, succeeds, and both services recover. No manual DB mutation required.Expedited P1 — deploy in the current window.
🤖 Generated with Claude Code