feat(ci): probe production federation — nothing was checking whether remotes load

[github-actions]

The portal shows 13 products and none of them renders its UI. Diagnosing that surfaced why it went unnoticed: no check in this repo ever asks production whether a federated remote actually loads.

check	what it actually covers
prod-smoke.yml	polls a health endpoint
post-prod-e2e.yml	drives user flows
e2e.yml	runs check-federated-assets.mjs — against a local preview, never prod

So the failure this shell is most prone to — an entry or chunk that 404s in production while every green check in the pipeline describes a different artifact — had no detector at all.

The portal was reporting the opposite of the truth

backend/src/routes/apps.ts:206:

// Accept any response (including 404) as long as the server responds
return response.status < 500 // Consider 2xx, 3xx, 4xx as healthy, 5xx as unhealthy

A remote whose assets 404 is reported healthy. That is not a gap in observability, it is observability asserting the opposite of reality, and it is why "13 products, none loading" looked like a working system.

Fixing that health check is a separate change — this PR builds the instrument that can prove what is actually being served, because I did not want to change the health semantics before having something that could measure the result.

What it does not assume

The serve path is a free variable (CLAUDE.md — "slug, display name, and the federated serve path are THREE INDEPENDENT questions"). Most repos publish /apps/<slug>/remoteEntry.js (Vite assetsDir: ''); FuzeFront's own fuzequality publishes /apps/<slug>/assets/remoteEntry.js.

So each app is probed at both candidates and the report names which one answered. The probe discovers the layout instead of re-asserting the convention it exists to check — an app answering at neither is the finding, not a probe bug.

Chunk verification delegates to the existing check-federated-assets.mjs, which already encodes the two things that make this real rather than a re-derivation of config: chunk specifiers resolve against remoteEntry.js's own URL, and a 200-with-HTML body is the 404 it really is.

Not vacuous — mutation-tested in both directions

A probe that can only fail is as useless as one that can only pass, so both were proven, not asserted:

scenario	result
empty slug list	exit 2 — a probe that checks nothing must never report success
unreachable host	exit 1, ❌ entry, both candidate paths named
entry + chunk both served as JS	exit 0, ✅
entry JS, chunk 200 + HTML	exit 1, ❌ chunks, "1 referenced — 0 loadable, 1 broken"

The last row is the one that matters: it is exactly the blank-panel failure, and a probe that only fetched the entry would have called it green.

Design notes

• Unauthenticated by design. It probes static asset paths, not GET /api/apps — that needs a token and is scoped by org membership + visibility (scopeAppsQuery), so it could not enumerate every app even with one.
• runs-on: ubuntu-latest, deliberately not the fuzefront ARC pool. A prod diagnostic has to be runnable exactly when the self-hosted pool is saturated, which is when an incident is most likely under investigation.
• PR trigger is path-scoped to the probe's own files. It hits live production, so it must not fire on every PR — but a detector whose first real execution happens after it merges is how you ship a broken detector.
• Daily schedule. A remote can stop loading with no commit here at all: a product repo redeploys, an ingress changes, a chart bumps an image. Catching that needs a clock, not a PR.

What this does and does not claim

It does not identify the root cause of the current outage — it is the instrument that can. What I established while building it, from the repos rather than from prod:

• Every remote's source now pins react: ^19.0.0, matching the host's requiredVersion. The React-18 singleton mismatch is fixed in source, so it is not the current explanation unless prod is running older images.
• Prod image tags are mixed: fuzeexecutive, fuzedeploy, fuzepicker, fuzex are SHA-pinned and current, while fuzesales, fuzecontact, fuzebi sit at 0.1.0 and fuzemarket at the mutable latest. Stale images explain some products but cannot explain all of them.
• fuzekeys is the one remote declaring a cross-origin remoteEntry (https://keys.prod.fuzefront.com/...) rather than a same-origin path.

The probe's first run on this PR is what turns that into an answer.

[izzywdev]

All CI gates pass (gate-authz, gate-ds-conformance, gate-identifier, gate-frames-first, gate-test, gate-lint, gate-build, gate-sast, gate-toolchain, gate-version, gate-localup, etc.). Approving per governance policy.