ARC capacity: 14 repos moved onto the shared node and saturated it

[izzywdev]

@claude — infra request. Raised from the consuming side; I have not touched anything in this repo's cluster config.

What happened

14 private repos were moved off ubuntu-latest onto their own ARC scale sets today, because the account's Actions budget refuses private-repo GitHub-hosted jobs outright (runner_id: 0, empty runner_name, no log — the job never starts). That migration is correct and working: jobs that had never run at all now run.

But it put 299 jobs across 14 repos onto the shared node at once, and it is saturated.

Evidence, from this repo's own tooling

cluster-query (runs-on: staging) is the cleanest measurement, because it's small, constant, and unrelated to the migrated repos:

dispatched	outcome
18:49 UTC	ran on staging-25mld-runner-z88sp, completed in 12s
19:29 UTC	queued 8+ minutes, runner_id: 0, still not started

Same workflow, same runner, same commit. The only variable is load.

This is not the queue-forever hazard from governance/ci-runners.md — the staging set exists and its listener is up. It's genuine contention.

Why it happened

runners/arc/ puts all 22 scale sets on one 4-vCPU / 8GB node (fuzeinfra-ci-runner-1) at maxRunners: 3 each. That's a ceiling of ~66 concurrent runner pods, plus unbounded dind sidecars, on 4 vCPUs. ci-runners.md says this plainly — "Capacity is shared, and it is small… the sum across sets is what matters, not the per-set number" — and notes the node has OOMed once before.

I read that and rolled all 14 anyway. That's on me; the fix on my side was to stop pushing, and the backlog does drain.

What would help, in rough order of value

1. More capacity — a second runner node, or a larger one. The fleet has roughly doubled its self-hosted footprint today and is not going back.
2. Bound the sum, not the per-set number. maxRunners: 3 × 22 sets is a ceiling nothing enforces against actual CPU. Lowering per-set maxRunners would trade latency for stability, which is the better failure mode than an OOM.
3. Cap the dind sidecars. They're currently unbounded, and they're the part most likely to take the node down rather than just slow it.

Not asking for

No change to scale-set names — the per-repo naming is what the consuming repos now declare in ci.runner, and the fuze-runner consolidation is a separate coordinated flip (see governance/ci-runners.md).

Related, and cheap

A concurrency group on the stamped workflows would cut a lot of this waste: a re-push currently stacks a whole new set of ~20 jobs instead of cancelling the superseded run. That's a FuzeSDLC change, not an infra one — I'll take it there once the current queue clears, since landing it means pushing to 14 repos again.

[izzywdev]

Sharper measurement, which changes the shape of the ask.

kubectl -n arc-runners get pods -o custom-columns=NAME,STATUS,NODE at 19:55 UTC:

fuzehub-sfxmj-runner-sqfbx           Pending   fuzeinfra-ci-runner-1
fuzehub-sfxmj-runner-64twq           Pending   <none>
fuzehub-sfxmj-runner-qmscs           Pending   <none>
fuzemarket-rpm2v-runner-49x97        Running   fuzeinfra-ci-runner-1
fuzemarket-rpm2v-runner-vqjsx        Pending   <none>
fuzemerchandize-7lf9r-runner-8ctv7   Pending   <none>
fuzepicker-tsphx-runner-77fg2        Pending   <none>
fuzeplan-kl544-runner-xk989          Running   fuzeinfra-ci-runner-1
fuzesales-sbx45-runner-ztxts         Running   fuzeinfra-ci-runner-1
fuzesdlc-7rvw9-runner-jpr5m          Running   fuzeinfra-ci-runner-1
fuzeservice-f6wl9-runner-dn4sl       Pending   <none>
fuzesocial-65pns-runner-2p6vz        Pending   <none>
staging-25mld-runner-ldcwz           Running   fuzeinfra-ci-runner-1
…

The binding constraint is not maxRunners. ARC is scaling up runner pods correctly — they just can't be scheduled. Pending with NODE: <none> means no node has capacity for them. Roughly 5 pods running against dozens pending.

So the earlier framing ("22 sets × maxRunners: 3 = a 66-pod ceiling") overstates what the cluster can actually deliver. The real ceiling is whatever fuzeinfra-ci-runner-1 can fit, which looks like ~5 concurrent runners plus their dind sidecars on 4 vCPU / 8 GB.

That reorders the three suggestions in the issue body:

1. A second node is the fix. Lowering per-set maxRunners would only stop ARC creating pods that were never schedulable — it would make the queue honest, not shorter.
2. Capping dind sidecars still matters for stability (they're the part most likely to take the node down rather than just fill it), but it won't add throughput.

Worth noting the queue does drain — a cluster-query job dispatched at 19:29 while saturated completed at 19:55. Nothing is wedged; it's throughput, not a fault.

---

Generated by Claude Code

[izzywdev]

Follow-up measurements, 20:12 → 23:42 UTC — and one pattern that is not just "the node is full"

The backlog does drain, but unevenly, and staging is the set that loses.

staging specifically is starved, while per-repo sets get runners

set	evidence
staging	cluster-query 32297269671 dispatched 20:12, got a runner at 23:01:40 — 2h49m. It then completed in 10 seconds.
staging	cluster-query 32300981563 released from the concurrency queue at 23:01:51, still runner_id: 0 at 23:42 — 40m and counting.
fuzedeploy	run 32296079334 created 19:59; jobs started 22:12, 22:19, 22:23, 22:25 on fuzedeploy-jdf99-runner-*.
fuzesales	run 32309510070 created 22:35; both jobs started within ~2 minutes on fuzesales-sbx45-runner-*.

So per-repo sets are being served — fuzesales got two runners inside two minutes at 22:37 — while staging waited hours across the same window. If that is scheduling/priority rather than raw capacity, it may be cheaper to fix than adding a node.

It matters more than the raw numbers suggest: cluster-query runs on staging, and it is the only read path a consuming repo has into the cluster. When staging starves, every product repo loses its ability to diagnose anything — tonight that blocked verifying a prod deploy (FuzeFront#749, merged 22:44) for over an hour. cluster-query also has concurrency: cluster-query, cancel-in-progress: false, so queries serialise: one starved slot stalls every subsequent query behind it.

Job-level starvation, not just queue depth

Within a single run, jobs are served a few at a time rather than together — fuzedeploy 32296079334 had 4 of 9 still queued after 2h45m while 5 had completed. Consistent with the 4-vCPU ceiling.

A large build can hold a slot for over half an hour

FuzeSales 32309510070 build-push (fuzesales, Dockerfile) — a whole-monorepo npm install — ran 32.6 minutes and then failed with npm error network at elapsed 1956.1s. Not 404, not 401, and the image already had its registry mapping, so this reads as I/O starvation rather than a config fault. Its sibling job building only one service's dependencies succeeded in 2m15s on the same pool. Worth weighing when sizing: one monorepo install can occupy a slot for half an hour and still fail.

On my side

I have stopped batch-pushing, and I have added a concurrency group with cancel-in-progress scoped to pull_request to FuzePlan's ci-cd.yml (a re-push there was stacking ~14 fresh jobs instead of superseding the previous run). Deliberately not applied to pushes to main, since cancelling a deploy run mid-flight could leave a half-published image set. The same gap exists in other repos' non-stamped workflows and I will land those staggered rather than in a batch.

No action requested beyond what is already in this issue — recording the staging-specific asymmetry because it points somewhere more specific than "add capacity", and the numbers above are what I would want if I were fixing it.

---

Generated by Claude Code

[izzywdev]

@fuze — re-routing this request through the LiteLLM gateway (was originally posted with @claude which hits the direct Anthropic path; using @fuze for gateway + fallback resilience).

The issue body stands: ARC runner capacity on fuzeinfra-ci-runner-1 is saturated. Primary ask is a second runner node or larger node. Secondary: staging set is being starved relative to per-repo sets, which blocks cluster-query for all consuming repos.

[izzywdev]

⚠️ Claude responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Anthropic API rejected preflight (HTTP 400): Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/32344237971

This notice is automated and will not retry on its own — the responder stops after one failure and waits. If this looks like an API-side problem (key revoked / out of credits / model or 1M-context beta not entitled / Anthropic outage), an operator should check the ANTHROPIC_API_KEY secret and fund/repair the Anthropic account first, then re-trigger by posting a new comment mentioning the responder.

[izzywdev]

Assessment (2026-08-24): Verify cluster state — likely resolved.

Since this issue was filed, durable capacity expanded (VLAN cutover + node reinstalls, see memory: durable-capacity-oom). The 14 repos that triggered this issue have been running on ARC scale sets backed by the durable node pool.

Verify with: kubectl -n arc-systems get pods and the ARC runners API (gh api repos/izzywdev/FuzeInfra/actions/runners) — if runner registration counts are healthy for each scale set, this is resolved. If ARC pods are still Pending on the CI node, the capacity issue persists.

Left open pending cluster verification.

[izzywdev]

Cluster health check (2026-08-25)

Ran kubectl get pods -n arc-systems -o wide via cluster-query (run 32794214713).

All 20+ repo listener pods: 1/1 Running. The acute saturation from the 14-repo migration has cleared — the backlog drained as expected.

Current runner pod inventory (arc-systems)

• Controller: arc-controller-gha-rs-controller — Running on vmi3396106 ✅
• Listeners (all Running, all on mendys-worker-1): fuzeagent, fuzebi, fuzecall, fuzecontact, fuzedeploy, fuzeexecutive, fuzefinance, fuzefront, fuzehub, fuzekeys, fuzemarket, fuzemerchandize, fuzepicker, fuzeplan, fuzesales, fuzesdlc, fuzeservice, fuzesocial, fuzex, mendysrobotics, mendysroboticswp, staging — 22 scale sets active
• Selfheal cronjob: 2 old Error pods (retained from prior CronJob runs, ~38h and ~2d21h) + 2 recent Completed runs. Normal.

Capacity questions (still open)

The acute saturation resolved, but the structural questions from the issue remain valid:

1. Second runner node — the fleet has doubled and shows no sign of shrinking back. Still worth doing.
2. Sum cap on maxRunners — 22 sets × 3 = 66 theoretical concurrent pods on 4 vCPUs is still the ceiling.
3. Concurrency groups — still a FuzeSDLC change (noted above).

These are medium-term capacity investments. Happy to track them as separate issues if useful — this one can be closed for the acute incident, or left open if you want to track the structural work here.

[izzywdev]

Fresh measurement, 2026-08-25 ~13:55Z — this is still live and now has a number attached.

Queue latency on the fuzeplan scale set is 2.5–4.5 hours. Measured from created_at → updated_at on runs that did eventually complete:

created	completed	latency	workflow
09:05:47Z	13:41:25Z	4h36m	Auto Merge PR
09:05:47Z	13:32:23Z	4h27m	Governance Sync
09:05:47Z	13:30:20Z	4h25m	MCP Maintain
08:55:17Z	11:53:07Z	2h58m	Governance Sync
08:56:42Z	10:09:38Z	1h13m	Auto Merge PR

The pool is not dead — I checked that hypothesis specifically and it is disproved; runners are being created and jobs are draining. It is oversubscribed, exactly as this issue says.

Consequence, and the reason I'm adding this rather than opening something new: FuzePlan#207 has 20 of 22 check runs still queued since 11:37Z, so a fix that is already pushed cannot be verified, and its PR cannot merge. Same shape on FuzeContact#59 — Build & push image queued since 12:47Z, though the fuzecontact pool is healthier (drains roughly one job per 3 minutes; 5 of 9 harden-gate jobs completed 13:24–13:32Z).

Two corrections to earlier characterisations of this, so nobody re-derives them:

1. This is not the GitHub Actions spending limit. I had attributed it that way earlier today. The jobs carry labels: ["fuzeplan"] / ["fuzecontact"] and are assigned to ARC runners (fuzecontact-v5kfg-runner-8pnv2, -k57sl) — self-hosted, not ubuntu-latest. The hosted-minutes budget is a separate and much smaller problem.
2. It is not uniform across the fleet. fuzesales completed 18/18 checks normally today. So this is per-scale-set capacity, not a global control-plane fault — which should narrow where to look.

No cluster action taken or requested from this side; raising it here because ARC is FuzeInfra's to operate.

---

Generated by Claude Code

[izzywdev]

Opened #675: bounds the dind sidecar's CPU/memory with a real resources block, so the k8s scheduler enforces an actual sum-wide ceiling on the pinned CI node instead of the notional per-set maxRunners number.

Verified current main state first — argocd/applications/arc-runners.yaml's 2026-08-21 maxRunners 3→5 raise (until 2026-08-30) is left untouched, as are all 22 scale-set names/URLs. Root cause traced directly in the pinned chart's Go templates: containerMode: dind (chart 0.14.2) injects the docker-in-docker sidecar with no resources block at all, and there's no values-only way to give it one while that shorthand is active — so the container that actually runs every job's docker build/compose/run was invisible to scheduler admission accounting. Fix: write the dind podspec out explicitly (same init container, same dockerd invocation — not a re-architecture) and give it requests: 500m/1Gi, limits: 2 vCPU/4Gi, matching the runner container's own envelope so a runner+dind pod pair reserves 1 vCPU/2Gi — a real ~4-pair ceiling on the 4 vCPU/8Gi node, summed across all scale sets.

This does not add a second/larger node (ask #1) — that's a cost/capacity call left for a human, not something to do unilaterally here. Leaving this issue open for that decision.

---

Generated by Claude Code

[izzywdev]

Opened #679 for ask #1 (capacity): raises TF_VAR_ci_worker_count 1 → 2 in terraform-plan-apply.yml, which is all that's needed — terraform/contabo/ci-workers.tf already provisions CI nodes generically via modules/contabo-k3s-node's range(var.ci_worker_count), so this adds fuzeinfra-ci-runner-2 on the same Contabo plan as fuzeinfra-ci-runner-1 (4 vCPU/8Gi), same region, no module changes.

The new node gets fuzeinfra.io/pool=ci + the fuzeinfra.io/ci=true:NoSchedule taint automatically at k3s agent registration (module cloud-init) — same as node 1. runners/arc/runner-scale-set-values.yaml already schedules every one of the ~22 scale sets by that label (nodeSelector/toleration, not node name), so no runners/arc/ changes are needed for the new node to start taking work.

This is a PR only — plan will post as a comment on #679 for review; merging is what triggers the actual terraform apply via this repo's merge-to-apply CD. Not applying/merging myself.

PR: #679

---

Generated by Claude Code