Deleted proxies leave their anchors behind

[ecv]

Summary

Deleting an HTTPProxy leaves its anchor behind on the Karmada hub, and everything the anchor owns stays alive and serving with it.

The anchor is the only thing tying a derived Gateway and HTTPRoute to an upstream object that lives on a different control plane, so nothing else can collect them. Each surviving Gateway then mints a Let's Encrypt Issuer, and one edge cluster has reached 902 Issuers driving 42 requests per second against a fleet baseline of 0.13.

One project namespace on the hub holds 145 anchors today, the oldest from 16 July.

Acceptance criteria

• An HTTPProxy deleted from a project control plane leaves no anchor on the hub
• An anchor whose upstream object is already gone is collected without anyone running a tool
• Anchors stranded before the fix are cleared

What is happening

Read on the Karmada hub on 2026-09-17 at 01:20Z, in ns-45e239ee-7a69-4e44-ba30-48f73f431101, project integration-24be19.

The hub still holds e2e-path-routing-2026-08-01t08-46-39z as both an HTTPRoute and a Gateway, created 1 August, with live ResourceBindings. The project control plane holds nothing from that date, so the upstream HTTPProxy is long gone.

The route carries a single ownerReference to a ConfigMap named anchor-f99c6b9b-907c-4e85-b1d5-d53544c95448, also created 1 August and still present. It carries no data. Its labels name what it stands for: meta.datumapis.com/upstream-cluster-name, upstream-group, upstream-kind, upstream-name and upstream-namespace.

Karmada is behaving correctly throughout. It propagates objects that genuinely exist, so the edge copies are right. The whole cascade is one delete away and that delete never happens.

Why nothing collects it

GatewayDownstreamGCReconciler deletes the anchor, but only for an object it observes going through a finalized deletion:

if dt := obj.GetDeletionTimestamp(); dt.IsZero() || !controllerutil.ContainsFinalizer(&obj, gatewayControllerGCFinalizer) {
	return ctrl.Result{}, nil
}

Miss that window, whether the controller is down, the finalizer is absent, or the event is dropped, and nothing revisits it. The reconciler is keyed on an upstream object that no longer exists, and the anchor carries no owner of its own, so Kubernetes garbage collection has nothing to follow either.

That is the same shape as #233, which proposed a periodic scan over owned resources and was closed in favour of one-off cleanups.

What a responder sees

Ten HTTPRouteStatusWarning alerts are firing in production against two of these routes across five sites, which is how this was found. They are accurate: the backendRef Services really are absent. The objects are broken and they are also immortal.

Related: #182 is the same failure on ResourceClaims, where deleting an HTTPProxy leaves quota consumed. https://github.com/datum-cloud/infra/issues/4684 records the Issuer and rate-limit cost.

[vvoytovych]

Edited 2026-09-23 after an independent re-check. Step 4 below is now marked as inferred, the fix note covers the Certificate path, the replacement is shown across runs, and the site count is four.

For the Gateways I traced, the leak is not a missed delete. The delete runs, and then another replica recreates the downstream Gateway. That is #319 on the Gateway path, as Mykola suspected there.

What I saw

Read in project integration-24be19 on 2026-09-23, with v0.28.1 in production. Two WAF test proxies were deleted five seconds apart. One Gateway was cleaned up, and the other is still on every edge.

For the leaked Gateway, one replica finalized it and deleted its anchor. Two other replicas then reconciled the same Gateway as if it were live:

12:30:56.572  kdbf9  finalizing gateway
12:30:56.572  kdbf9  deleting anchor for upstream gateway
12:30:56.600  4pp4q  reconciling gateway
12:30:56.617  4pp4q  downstream gateway not yet accepted/programmed, requeueing
12:30:56.619  kwvxb  reconciling gateway
12:30:56.620  kwvxb  downstream gateway not yet accepted/programmed, requeueing
12:30:57.047  4pp4q  Reconciler error  gateways.gateway.networking.k8s.io "e2e-waf-observe-2026-09-23t12-27-59z" not found

These are selected lines from Loki for this one Gateway. I cut the level and the JSON context, and show each replica by its pod suffix.

The not yet accepted/programmed lines at 12:30:56.6 carry ipv4=None, accepted=false and programmed=false. The same lines appear from 12:29:31Z while the first downstream Gateway comes up, and stop at 12:30:05Z once it is programmed. So at 12:30:56.6 the downstream Gateway is new and empty.

For the Gateway that was cleaned up, all three replicas took the finalize path. None of them reconciled it as live afterwards.

The edge shows the replacement. On all 18 edges, the leaked Gateway's creation time moved from 12:29:34Z to 12:30:57Z. The HTTPProxy was deleted at 12:30:52Z.

It is not one run. On us-central-1-alice, none of the 158 e2e Gateways cleaned up in the last two days was ever replaced. Four of the six leftovers from the same period were.

Why the downstream Gateway comes back

Read at 069bb2a. None of these paths has changed since v0.28.1.

1. finalizeGateway deletes anchor-<uid> (gateway_controller.go#L1787).
2. A replica that reads the Gateway without its deletion timestamp logs reconciling gateway and runs ensureDownstreamGateway (gateway_controller.go#L165-L193).
3. With no downstream Gateway present, ensureDownstreamGateway creates one and sets its controller reference (gateway_controller.go#L339-L340).
4. SetControllerReference creates the anchor when it is missing (mappednamespace.go#L173-L183). The same live path also calls it with the upstream Gateway as owner for a new Certificate (gateway_controller.go#L985).
5. The upstream Gateway is then gone, so no replica finalizes what was created.

Step 4 is my inference. I have not read the anchor on the hub.

One detail for the fix. Gateways carry gatewayControllerFinalizer (gateway_controller.go#L205), not gatewayControllerGCFinalizer. Their downstream cleanup is finalizeGateway, not GatewayDownstreamGCReconciler.

What I'd ask

• One read on the hub would settle step 4. The anchor-<uid> for this Gateway should have a creation time near 12:30:56.6Z. The downstream Gateway's owner reference should point at it. I cannot read the hub, so could someone who can check it?
• The live path should not create downstream objects for an owner that is gone or terminating. A live read of the owner narrows the window but cannot close it. The Sharded coordinator does not prevent multiple replicas from reconciling the same project #319 fix closes it.
• The Sharded coordinator does not prevent multiple replicas from reconciling the same project #319 fix should cover gateway_controller.go, as Mykola noted there.
• The periodic sweep in this issue's acceptance criteria is still needed for the leftovers.

Each leftover is one more Gateway in the full rebuild that #381 describes. On 2026-09-23 the edge gateway controller was OOM-killed at its 3 GiB limit on four sites.

To see the lines above in Loki, over 2026-09-23 12:27Z to 12:45Z:

{namespace="datum-system", container="manager"} |~ "e2e-waf-(observe|enforce)-2026-09-23t12-27-59z"

To count the leftovers on each edge:

count by (cluster) (gatewayapi_gateway_info{name=~"e2e-.*"})

[scotwells]

@vvoytovych agreed these findings are likely also fixed by #320