Skip to content

feat: Decide internet egress per network, not per node - #580

Draft
scotwells wants to merge 1 commit into
mainfrom
feat/per-network-egress-route
Draft

scotwells wants to merge 1 commit into
mainfrom
feat/per-network-egress-route

Conversation

@scotwells

@scotwells scotwells commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Whether a tenant network reaches the internet is decided per node today. One list of translating shards is set at install time, so every network on that node routes toward it and a network that should have no outbound path gets one anyway.

This makes the decision per network: a network reaches the internet when its attachment declares it, reaches nothing when it does not, and loses an already installed route when the declaration goes away.

The declaration names no shard, so the list a node already holds decides where its networks leave, and each attachment decides whether.

API

// One attachment's stanza in a node's CNI config. Two answers.

// reaches the internet, through the shard this node's configuration names
{ "type": "galactic-bgp", "vpcattachment": "web-01", "egress": { "internet": { "mode": "Enabled" } } }

// reaches nothing outside the platform, and withdraws a route that is there
{ "type": "galactic-bgp", "vpcattachment": "web-01" }

Important

An attachment whose stanza carries no declaration gets no egress route, and one that declares egress on a node naming no shard fails its attach. Every conflist the lab renders now carries the declaration, and a deployment rolling this out has to render it before nodes can rely on the node-wide list alone.

Test plan

  • A network with no egress declared installs no outbound route, on a node that has shards
  • A network that loses its declaration has both its routes withdrawn
  • A network that declares egress on a node naming no shard fails its attach with a reason
  • Every lab tenant keeps the egress it has today

Related to datum-cloud/network-services-operator#475

A node holds one list of egress shards and routes every network on it
toward the first reachable entry, so a network that declared no egress
gets a default route out anyway. The attachment's own stanza now carries
the declaration, and the node acts on it.

The declaration names no shard. The node routes toward the shard its
own configuration names, so the list decides where and each attachment
decides whether. A network that declares egress on a node naming no
shard fails its attach, so a node an operator forgot surfaces at the
first instance rather than behind an attach that succeeded without
egress.

Key changes:
- Add the egress declaration to the galactic-bgp stanza
- Install the VRF's egress routes only for a declaration of Enabled,
  and withdraw them otherwise, touching no bpffs on a node whose
  datapath has not loaded
- Fail the attach when egress is declared and the node names no shard
- Declare egress on every lab tenant so the lab keeps what it has
- Document the declaration beside the node-wide list it gates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@scotwells
scotwells force-pushed the feat/per-network-egress-route branch from aa405ed to 5472d11 Compare September 24, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant