Member pods can reach inference backends directly, bypassing LiteLLM

[sre-helmcode]

Found by the Security reviewer during helmcode/nan-devops#389. Not introduced by that PR — the path predates it — but that PR doubles the number of nodes exposing it, so it should not stay unfiled.

The gap

The allow-internet-egress NetworkPolicy applied to member namespaces excludes 10.244/16, 10.245/16, 127/8, 10.8.0.0/24 and 172.16/12 from its egress CIDR — but not 10.10.0.0/24, the node VLAN. Verified identical across a sample of the member namespaces.

The host firewall accepts from 10.244/16 and 10.245/16 (pod CIDRs), and the inference backends answer an unauthenticated GET. So a pod in a member namespace can reach:

• 10.10.0.9:8090 and 10.10.0.5:8090 (whisper)
• 10.10.0.9:8880 and 10.10.0.5:8880 (kokoro)
• vLLM :8001, TEI :8080 on the GPU nodes

directly — no API key, no rate limit, no SpendLogs row, no monthly cap. Every guard we have lives in LiteLLM, and this path does not go through it.

Why it matters more now

Before nan#86 the path existed against eu009. It now exists against eu005 too, because the second audio backend is deployed the same way. The fix does not get harder with two nodes, but the blast radius grows with each backend added this way.

Fix

Add 10.10.0.0/24 to the except list of the egress rule in the NetworkPolicy template (cloud-api generates it per namespace). One line in the template.

It is not a one-line rollout: it applies to ~585 existing member namespaces, so it needs the usual care — template change, verify on a fresh namespace, then a controlled sweep of the existing ones, watching for anything that legitimately talks to the VLAN.

Worth checking at the same time

Whether the backends should answer an unauthenticated request at all. They are OpenAI-shaped servers with no auth (speaches, Kokoro-FastAPI, vLLM), which is why the network boundary is the only control.

Context: helmcode/nan-devops#389, project_nan_host_firewall_2026_08_19.

[sre-helmcode]

Remediation complete — summary of what shipped.

Confirmed exploitable before fixing, not just reported: from a pod carrying the member policy, GET http://10.10.0.9:8090/v1/models returned 200 with the model list.

829 namespaces	swept to the RFC1918 except list — 632 vulnerable → 0
nan-cloud-api#212	the Go generator, merged and deployed — verified the running binary contains 10.0.0.0/8 and no longer 10.244.0.0/16
nan-devops#544	the second copy of the same policy (k8s/templates/networkpolicy.yaml, applied by create-member.sh) — found by the QA reviewer; unfixed it would have undone the sweep
link-local	169.254.0.0/16 added to both generators: measured, the policy was allowing it and only the absence of a listener stopped it

The fix ports the shape internal/spaces/resources.go has had since Spaces shipped: whole-RFC1918 except list, plus one rule opening the VLAN for TCP 30000-32767 only (the marketplace tenant-DB NodePorts, which resolve to VLAN addresses since the host firewall dropped NodePorts on the public NIC, nan-devops#384). The backends listen below that range.

Verified after the fix:

whisper 10.10.0.9:8090    http=000 in 0.0005s   (policy rejection, not timeout)
kokoro  10.10.0.5:8880    http=000 in 0.0002s
metadata 169.254.169.254  http=000 in 0.0002s   (was a 7s timeout: allowed, just unanswered)
internet 1.1.1.1          http=301 in 0.05s     (unaffected)

The sweep alone was not enough, and the QA review was right to insist. Production ran pre-fix code throughout, so two namespaces were minted vulnerable while remediating — member-juanoliver during the sweep and member-vicent-sd a minute before the audit that caught it. Both are closed; the deploy is what stopped the leak.

Reviewers cleared the adjacent escape routes: the node's public IP does not reach the backends (they bind to the VLAN only), a member cannot create Endpoints to DNAT around the carve-out (nan-member grants services, not endpoints), hostNetwork is blocked by baseline PodSecurity, and the cluster is IPv4 single-stack.

Two follow-ups worth their own issue

1. The carve-out is by port range, not by service. Every NodePort is reachable from every member pod — today six tenant databases (credential-protected) and ssh-gateway. A future NodePort on something sensitive would re-open a path silently. Noted in the code and KUBERNETES.md.
2. The backends answer an unauthenticated GET at all — speaches, Kokoro-FastAPI and vLLM ship no auth, which is why the network boundary is the only control.

Also out of scope here: the host firewall accepts everything from the pod CIDR toward the node itself, so wildcard-bound services (kubelet :10250, apiserver :6443, sshd) remain reachable from a member pod via the node's public IP. All authenticated, and the old policy allowed the same — but it is the same class.