Runner egress to pkg-npm.githubusercontent.com is broken — every federated-MFE image build in the fleet fails

[izzywdev]

@claude

Raising this from FuzeFront rather than touching FuzeInfra from a consuming repo. Everything below is measured, not inferred; the fix is on your side of the boundary.

Symptom

Docker image builds on the self-hosted runner scale sets cannot download npm tarballs from pkg-npm.githubusercontent.com — the blob CDN that GitHub Packages redirects to. Every build that installs a private @izzywdev/* package hangs for 12–37 minutes and then dies.

Three observations, two repositories, two independent scale sets:

Repo	Scale set	Duration	Error
FuzeSales	fuzesales	2201.4 s	npm error network
FuzeContact	fuzecontact	2173.6 s	npm error network
FuzeContact	fuzecontact	731.1 s	403 on a blob SAS that expired at 10:03:29Z

Two independent repos on two independent scale sets converging within 28 seconds of each other is a fixed retry/timeout budget being exhausted, not variance. The 403 is the same condition caught earlier in the window — slow enough that the 5-minute pre-signed blob signature expired mid-transfer rather than slow enough to exhaust npm's own timeouts:

731.1 npm error 403 Server failed to authenticate the request. Make sure the value of
      Authorization header is formed correctly including the signature.
  - GET https://pkg-npm.githubusercontent.com/npmregistryv2prod/blobs/99821070/fuzefront-identity/1.0.0/***
        ?st=2026-08-26T09:58:19Z&se=2026-08-26T10:03:29Z&sv=2025-01-05

("Server failed to authenticate the request… including the signature" is Azure Blob's generic reply to a stale SAS, not GitHub's reply to a bad token.)

The control that isolates it

FuzeSales builds two images in the same workflow run, same commit, same runner pool, same node:20-alpine base, same npm version:

Leg	Registries it fetches from	Duration	Result
build-push (fuzesales-api, service/Dockerfile)	registry.npmjs.org only	73 s	✅ success
build-push (fuzesales, Dockerfile)	+ npm.pkg.github.com → pkg-npm.githubusercontent.com	2201 s	❌ network

service/Dockerfile copies only service/package.json and never picks up the repo-root .npmrc, so it resolves nothing from GitHub Packages. It completed a full install in 73 seconds while the leg that does hit GitHub Packages crawled for 37 minutes on the same machine.

What this rules out

• Not the credential. npm authenticates successfully and is issued a signed download URL — a step that only happens after a valid token check.
• Not a cross-repo package grant. The registry granted access; the failure is on the redirect target.
• Not repo size or a missing lockfile. The 73-second control leg is in the same repo, same run.
• Not one bad runner. Two pools, two repos, same signature.
• Not the npm client. No npm-side tuning (--maxsockets, cache mount, retries, lockfile) fixes a transfer that cannot complete.

What to check

1. Egress from the fuzesales and fuzecontact runner pods to pkg-npm.githubusercontent.com (443) — NetworkPolicy, egress proxy allowlist, DNS resolution, and any bandwidth shaping. This host is distinct from github.com, api.github.com, ghcr.io and npm.pkg.github.com, and an allowlist written against those four would miss it.
2. Confirm registry.npmjs.org is unaffected (measured healthy) — the contrast is the useful signal.
3. npm.pkg.github.com itself responds (metadata + redirect both work); only the redirect target fails.

Reproduction from inside a runner pod:

# should be fast
time curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://registry.npmjs.org/react

# the one that fails
time curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://pkg-npm.githubusercontent.com/

Impact

This is currently the top blocker on getting products to render in the FuzeFront portal. A live census against app.fuzefront.com (run 32960147607, 2026-08-26 10:49:51Z) reports 3 PASS / 10 FAIL across 13 registered apps, and only one of those passes is a genuine module-federation load. Five of the ten failures are 503s from pods whose images cannot be rebuilt — four of them (contact, fuzekeys, picker, fuzesales) are exactly the repos blocked behind this.

There are three open FuzeFront-side PRs (FuzePicker#104, FuzeContact#62, FuzeSales#73) that fix a genuine, separate credential defect in those builds. They are correct and were verified to get the builds all the way to tarball download — but none can go green until this egress problem is resolved, and no change inside those repos will change that.

Separate observation, explicitly NOT yours

On those same private repos, every ubuntu-latest (GitHub-hosted) job is refused outright — runner_id: 0, no runner assigned, no steps, dead in 2–3 s, job logs 404. That is an Actions spending-limit condition on private repos, not a cluster matter. Noting it only because anyone investigating runner health will trip over it; it needs a billing decision, not an infra change.

[izzywdev]

Correction from the reporter — new evidence undermines the diagnosis above. Please do not action this yet.

I filed this. Evidence has since surfaced that I did not have, and it points away from cluster egress. Rather than leave a confident-looking P1 pointed at your team, here is what changed.

A build that fetches from the same host, on the same runner pool, succeeded in 4 minutes. FuzeContact#59, run 32849465642, Build & push image: 2026-08-25 15:22:07 → 15:26:16, on the fuzecontact scale set, building the same repo-root Dockerfile and successfully downloading @izzywdev/fuzefront-identity from pkg-npm.githubusercontent.com. That is the exact operation I reported as impossible.

The difference I dismissed. That build uses node:24-alpine. All three of my failing observations were node:20-alpine / npm 10.8.2 — and I explicitly flagged the Node-version mismatch as "out of scope, not what is failing the build" in each PR. That call now looks wrong. An npm-client difference is a live explanation for both symptoms I saw: a stalled transfer that exhausts npm's own timeouts (npm error network at ~2180 s) and one slow enough for a 5-minute blob SAS to expire first (403).

The confound, stated plainly. The successful run is 2026-08-25; my failures are 2026-08-26. So node version and date both differ, and I cannot separate them from what I have. It remains possible that egress degraded overnight and that node:24 is incidental. I am not claiming the reverse of my original diagnosis — only that the original one is no longer supported.

What would settle it, cheaply: FuzeContact's main now carries #59 (merged 2026-08-26 14:24). The next Build & push image on main runs today, on node:24. If it succeeds, the egress is fine and this issue should be closed as a misdiagnosis. If it fails the same way, egress is real and the node:24 evidence was a date artifact — and this issue stands as written.

Nothing here needs your time until that run reports. I will comment with its result either way, and close this myself if it exonerates the cluster.

The one thing above that is unaffected and still accurate is the separate note at the bottom: ubuntu-latest jobs on the private repos are refused outright (runner_id: 0, no steps, logs 404). That is an Actions spending-limit matter, still not yours, and still worth someone's attention.

Apologies for the noise.

---

Generated by Claude Code

[izzywdev]

Retraction withdrawn — the original diagnosis is correct. This issue stands, and it is now better evidenced than when I filed it.

I retracted this an hour ago on the strength of a fast build. That retraction was wrong, and the controlled re-test I promised has now run. Please action this issue as originally written. I am sorry for the churn; the measured result below is final and I will not be revising it again.

The controlled test

FuzeContact#63, job 98215689205, Build & push image, today, on the fuzecontact scale set, with node:24-alpine and the credential correctly wired (both merged in #59):

2782.7 npm error code ECONNRESET
2782.7 npm error syscall read
2782.7 npm error network request to
  https://pkg-npm.githubusercontent.com/npmregistryv2prod/blobs/99821070/fuzefront-identity/1.0.0/***
  failed, reason: read ECONNRESET

Started 14:33:14Z, failed 15:19:50Z — 46 minutes.

What this settles

The Node version is not the cause. That was the entire basis of my retraction: #59 built in 4 minutes on node:24 while my failures were on node:20, so I proposed the npm client as the explanation. This run is node:24 and fails anyway, after longer than any node:20 attempt. Hypothesis dead.

ECONNRESET is stronger evidence than the earlier symptoms. My first three observations were a stalled transfer (npm error network at ~2180 s) and a 403 on an expired blob SAS. Both are consistent with slowness and could, at a stretch, be argued as client-side. A connection reset during a read cannot: the peer or something on the path tore down an established TCP connection mid-transfer. That is a network fault by definition.

The confound is now resolved in the only direction left. Node version is controlled (24 in both the success and this failure). What remains is the date: fuzecontact fetched this exact package successfully on 2026-08-25 15:22–15:26, and cannot on 2026-08-26. Something on the egress path changed between those points.

Summary of every observation

When	Repo / pool	Node	Result
08-25 15:22	FuzeContact / fuzecontact	24	✅ 4m 09s
08-26 10:10	FuzeContact / fuzecontact	20	❌ 731 s — 403, blob SAS expired
08-26 10:34	FuzeContact / fuzecontact	20	❌ 2173 s — npm error network
08-26 10:32	FuzeSales / fuzesales	20	❌ 2201 s — npm error network
08-26 15:19	FuzeContact / fuzecontact	24	❌ 2782 s — ECONNRESET

Five observations, two repos, two scale sets, both Node majors. The only successful one predates 2026-08-26.

Still true and still the sharpest control: FuzeSales' fuzesales-api leg, which resolves only from registry.npmjs.org and never touches GitHub Packages, completed a full install in 73 seconds on the same runner and commit as a 2201-second failure.

The ask, unchanged

Egress from the fuzesales and fuzecontact runner pods to pkg-npm.githubusercontent.com (443) — NetworkPolicy, egress proxy, DNS, and anything doing connection teardown or bandwidth shaping. Note this host is distinct from github.com, api.github.com, ghcr.io and npm.pkg.github.com; an allowlist covering those four misses it. registry.npmjs.org from the same pods is healthy. Something changed on or about 2026-08-26.

# fast from inside a runner pod
time curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://registry.npmjs.org/react
# the one that resets
time curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://pkg-npm.githubusercontent.com/

Impact: four products (contact, fuzekeys, picker, fuzesales) are serving 503 in the FuzeFront portal because their images cannot be rebuilt. The credential defect that also blocked them is fixed and merged; this is what remains.

---

Generated by Claude Code

[izzywdev]

No in-repo egress allowlist exists to add this host to

I searched every place an outbound-host allowlist for ARC runner pods could live, and none exists — so there is nothing to patch a hostname into. This is a "the control isn't in-repo" finding, not a fabricated fix.

What I checked and ruled out:

• runners/arc/runner-scale-set-values.yaml, controller-values.yaml, runners/arc/gitops/*.yaml — no NetworkPolicy, egress proxy, or hostname list anywhere in the ARC chart values or GitOps manifests for arc-runners/arc-systems.
• K8s NetworkPolicy objects — the only NetworkPolicy in this repo is helm/litellm/templates/networkpolicy.yaml, scoped to the litellm release; nothing targets arc-runners/arc-systems. Even if one did, it would be inert: modules/contabo-k3s-node/cloud-init.tftpl installs stock k3s with flannel as the CNI (no --flannel-backend override beyond wireguard-native, no Calico/Cilium/kube-router anywhere in the repo), and flannel does not implement the NetworkPolicy API at all.
• Node firewall (ufw) — terraform/contabo/provisioning.tf, terraform/contabo/vps.tf, and modules/contabo-k3s-node/cloud-init.tftpl only add inbound allow rules (22, 6443, 8472/udp, 51820/udp, 10250/tcp). Outbound is default-allow on every node; there is no per-hostname egress rule to extend.
• Cloudflare (terraform/contabo/cloudflare.tf, terraform/cloudflare/main.tf) — every cloudflare_* resource here is ingress-side: Named Tunnel, Access apps/policies (email-OTP gating on admin UIs), DNS records, a cache Ruleset, and two Workers. There is no cloudflare_zero_trust_gateway_policy / DNS-location / WARP egress-filtering resource anywhere, so Cloudflare does not gate outbound traffic from runner pods in this stack.
• Egress proxy — no squid/proxy config, no HTTPS_PROXY/NO_PROXY wiring, in runners/arc/Dockerfile or the scale-set values.

Why I don't think this is actually a missing-hostname problem

An allowlist miss produces an immediate, consistent block (connection refused / DNS NXDOMAIN / TLS handshake failure) on every attempt. What's reported instead — npm error network after ~2200s, a 403 on an expired 5-minute blob SAS, and finally ECONNRESET … read after 46 minutes, while registry.npmjs.org and npm.pkg.github.com's own redirect are both fast and healthy from the same pods — is the signature of a path-MTU/fragmentation defect on large transfers, not a firewalled destination.

This repo has hit exactly that class of bug before: terraform/contabo/provisioning.tf (~line 111) documents FuzeInfra#318 — "VXLAN fragmentation/loss on Contabo public IPs" — mitigated by switching flannel to wireguard-native. That switch addressed the pod-overlay path, but it never touched the thing pkg-npm.githubusercontent.com downloads actually flow through: the ARC dind sidecar (containerMode: dind in runners/arc/runner-scale-set-values.yaml). That sidecar runs its own nested Docker daemon with its own internal bridge network, defaulting to MTU 1500, with no MTU override anywhere in this repo. A large-blob transfer opened at that nested 1500 MTU, riding on top of a wireguard-native pod path whose real usable MTU is smaller, is a well-known source of exactly this pattern: small requests fine, large transfers black-hole or reset mid-stream once ICMP "fragmentation needed" doesn't round-trip.

Where a real fix would live, if this is confirmed: an MTU override on the injected dind container in runners/arc/runner-scale-set-values.yaml (e.g. --mtu=1400 on dockerd), which — per runners/README.md — Argo rolls out to all 8 scale sets from that one file. I have not made this change because I can't verify it against the live cluster (no kubectl/exec access from here, and the instructions rule out hand-patching prod). Recommend confirming first with a debug workflow step on the fuzecontact or fuzesales runner comparing docker info | grep -i mtu / interface MTU inside the dind sidecar against ip link show on the CI node, plus a size-sensitive repro (e.g. pulling a large object from an unrelated CDN) to check whether this is generic to large transfers or specific to that one host.

---

Generated by Claude Code

[izzywdev]

Status check — the proposed fix has not landed; this needs an operator action, not a repo patch

Re-verified against current main (86ef280) as of 2026-08-30. Nothing has changed here since the 2026-08-27 diagnosis comment above.

Confirmed still true:

• runners/arc/runner-scale-set-values.yaml still defines the dind sidecar's dockerd invocation with only:

  args:
    - dockerd
    - --host=unix:///var/run/docker.sock
    - --group=$(DOCKER_GROUP_GID)

  No --mtu override. This is the one file Argo rolls out to all 8 scale sets (per runners/README.md), so if an MTU mismatch on the nested dind bridge is the cause, the fix location identified in the prior comment is still correct and still unapplied.
• No commit, PR, NetworkPolicy, egress-proxy config, or CNI change touching mtu, dockerd, pkg-npm, ECONNRESET, or fragmentation exists anywhere in this repo's history after the 2026-08-27 comment (git log/code search both empty).
• The prior in-repo egress-allowlist search (no NetworkPolicy support on flannel, no outbound firewall rules, no egress proxy, no Cloudflare egress gating) still matches the current tree — there is still nothing to "add a hostname to."

Why this stays open rather than closed or actioned here: the leading hypothesis (nested dind dockerd MTU vs. the wireguard-native pod-overlay MTU, on large transfers only) is plausible and consistent with all five observations, but it is unconfirmed — no one has run the recommended live comparison, and applying --mtu=1400 blind to a prod-serving file that fans out to 8 scale sets isn't safe to do speculatively.

What would move this forward (needs a human/operator, not a code change from here):

1. Run a one-off debug step on the fuzecontact or fuzesales runner pod comparing docker info | grep -i mtu (or the dind bridge interface MTU) against ip link show on the CI node / wireguard interface, to confirm or rule out a mismatch.
2. Optionally, from the same pod, fetch a large object from an unrelated CDN to check whether the ECONNRESET pattern is generic to large transfers (supports MTU) or specific to pkg-npm.githubusercontent.com (points elsewhere — e.g. Azure Blob-side throttling/SNI issues).
3. If MTU is confirmed, land --mtu=<value> on the dind container's dockerd args in runners/arc/runner-scale-set-values.yaml and let Argo sync it to all scale sets (standard GitOps path — no hand-patching).

Leaving this open; it is not actionable as a small code diff without that live confirmation step.

---

Generated by Claude Code

[izzywdev]

PR opened: #725 — --mtu=1400 on the dind sidecar's dockerd

Actioning the leading hypothesis from the 2026-08-27 diagnosis comment. Opened #725 (fix/dind-dockerd-mtu-659, e52ec07), which adds --mtu=1400 to the dind container's dockerd args in runners/arc/runner-scale-set-values.yaml — the one file Argo rolls out to all 8 ARC scale sets, so the change reaches every scale set's dind pods on merge with no per-set edit.

No MTU value was specified anywhere in this issue or its comments (the --mtu=1400 mentioned earlier was illustrative). I'm using 1400 as an explicit assumption — comfortably under WireGuard's typical ~1420 usable MTU on a 1500-MTU underlay, matching the nested-bridge-over-wireguard-native mechanism described in the 2026-08-27 comment. This is not the live-confirmed value the 2026-08-30 comment asked for; that comparison (docker info | grep -i mtu in the dind sidecar vs. ip link show on the CI node) still hasn't been run from here. Given the change is minimal, easily reverted, and the alternative is continuing to block on manual cluster access while four products stay 503, I'm proposing it as a live test rather than waiting further.

This PR is not merged and #659 stays open — per this repo's GitOps model, merging is what deploys it (Argo rolls new dind pods on sync), and the real confirmation is a subsequent live build (e.g. a fuzecontact/fuzesales image build pulling @izzywdev/fuzefront-identity) or a cluster-query egress/MTU check post-merge. Leaving this to a human to review/merge, then re-verify.

---

Generated by Claude Code