Conversation
Nothing created an egress shard object. Every shard in existence was hand-written next to a per-node DaemonSet overlay in a container lab, so a cell's egress tier existed only where someone had typed it out, and the pool label the shard type exports had nothing stamping it for a class's selector to read. An operator now declares the pool once per cell: the namespace its shards live in, the nodes commissioned into it, and the labels a class selects on. The builder in the cell expands that into one shard per selected node. Commissioning stays an operator's act and provisioning is not demand-driven. A shard is unusable until its SRv6 identifier exists, that identifier is still operator-supplied process configuration on the node with no allocator behind it and a documented collision hazard, so a shard created in response to a network's demand would attach, translate nothing, and then be skipped by the very selector meant to find it. Key changes: - Add EgressShardPool, cluster-scoped beside EgressShardParameters, whose schema refuses an empty node selector and refuses to stamp the family labels that restate an address assignment a pool does not make - Add a node opt-in label, because the label every translating node carries is compute's own role label and selecting on it would build a shard per compute node, making the egress address per-node - Build shards in name order with the pool label set from the pool's own name, so an operator's labels cannot point it elsewhere - Create and never rewrite: a shard's address is write-once and its labels are what moves traffic, so an existing shard is left as found and the builder's shard access gains create alone - Leave a shard alone when its pool is deleted or its node stops matching, rather than stranding the return traffic of every flow it translates - Report a pool that selected no node, which otherwise looks exactly like a working one until a network asks to egress through it - Repin the network module to its branch head Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A location's egress was selected twice from the same inputs: once when the node was told which shards to route toward, and again when a consumer was told which address their traffic leaves on. The two disagreed. The reported address came from the first shard by name, while the node installs the first shard whose SID it can resolve a route toward, and a node that is itself a shard can never resolve a route to its own advertised SID — every compute node runs the translator. A workload on such a node read one source address and egressed from another, silently breaking any allow-list built on it. A claim now records the binding, made once. One claim per network context that declares egress, so the binding outlives the instances using it: an interface is replaced routinely, and a binding that followed one would move a network's source address every time that happened. The shard holds no list of the networks it serves. Which networks a shard serves is answered by listing the claims labelled with its name, the way the network and location labels already make counting a presence's consumers a list query, and the only state a binder adds to a shard is one finalizer while a claim is bound. Key changes: - Add EgressShardClaim, whose spec carries the resolved terms verbatim and is immutable, and whose status holds the binding — following SubnetClaim, which records its allocation the same way, rather than the interface claim's reference on the provisioned object, which only a one-to-one binding can use - Bind in the cell's single writer, refusing a candidate with a named reason per incompatibility so a claim that bound nothing says what stopped it - Write the readiness reasons that were defined with no writer: Ready, and AddressUnavailable for a bound shard whose address is unclaimed, and Unavailable for a location with no usable shard - Hold a bound shard open with a finalizer, and release it when its consumer set empties, so decommissioning is an act someone takes rather than an outcome networks discover - Read the binding where egress used to be selected a second time, keeping the parameters guard that lets another implementation serve a class, and keeping the node's candidate list an ordered array so recording a standby shard later needs no node change - Narrow the shard event handler from every interface in the cell to the interfaces of the networks bound to that shard - Carry the class's sharing onto the claim and branch on nothing: dedicated capacity is a hand-commissioned node no controller can grow while nothing allocates a shard's identifier Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Translation runs on the node an instance attaches to, so nothing selects a shard. The claim becomes a record: one per attachment, owned by it, naming the node the attachment landed on and the families its network declared. The binder resolves the shard on that node and writes it to status, and reports egress readiness on the attachment. The record decides nothing. The node routes toward its own shard from the moment the attachment exists, and the record follows once the attachment has reported where it landed. Key changes: - Reshape EgressShardClaim to attachment, nodeName, and families - Bind by node reference, refusing a shard that is not ready, whose spec and status disagree, or that serves no declared family - Report InternetEgressReady on the attachment, patched so the controller rendering it is not overwritten - Replace a record when its attachment moves nodes - Remove EgressShardPool, its controller, CRD, and tests - Merge the attachment branch that carries the declaration to the node Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every compute node runs its own egress shard, and the node routes toward it the moment an attachment exists, but nothing writes down which shard an attachment ended up on or tells a consumer when a node has none.
This records one claim per attachment, naming the node it landed on, and resolves the shard on that node into the claim's status once the attachment has reported where it is.
The record decides nothing, so egress is live when the attach completes and the claim follows. It exists so the binding is readable, so a node without a usable shard produces a condition a consumer sees on their interface, and so a later tier that does select among shards binds through the same object.
API
The attachment gains an
InternetEgressReadycondition written from the same pass, which is what the interface a consumer holds reads.Test plan
Related to datum-cloud/network-services-operator#475
🤖 Generated with Claude Code