Connector status changes rebuild gateway config at every edge site

[ecv]

Summary

Every change to a Connector's status, however small, makes the gateway controller at every edge location rebuild its whole configuration.

From 13:12Z on 11 September to 00:45Z on 13 September, Connector status changed 150 to 280 times an hour, so every edge location rebuilt at that rate, at 6 to 15 seconds each.

Edge locations should rebuild only when something they route on changes, and one unstable Connector should not drive work at every location.

What success looks like

• A status change that leaves online state and tunnel address unchanged causes no rebuild
• Repeated status churn on one Connector causes a bounded number of edge rebuilds
• A newly connected or disconnected tunnel still reaches the edge as quickly as it does today

Related to https://github.com/datum-cloud/infra/issues/5303 and https://github.com/datum-cloud/infra/issues/5339

[ecv]

Onset. Connector status writes in the project control planes rose at 13:12Z on 11 September, in the same minute for every edge location. In six-hour windows they ran between 0 and 220 (typically 50 to 130) from 9 to 11 September, then 1150 to 1570 from 18:00Z on the 11th onward. The hub writes to Connectors follow them almost one for one (414, 1262, 1321, 1151, 1524 against 668, 1308, 1431, 1148, 1571 in the same windows), and each hub write reaches every edge, where the gateway controller watches Connector as an extension resource and rebuilds its full configuration. Edge locations now see 150 to 190 rebuilds an hour, each taking 13 to 15 seconds where they took 5 to 7.

Why it is not a release. Production Flux stayed on infra v0.83.41 until about 14:50Z, and NSO has run v0.26.4 since 7 September. The infra v0.83.42 rollout of the iroh gateway v0.1.4 image reached stable sites at 14:45Z and was rolled back in v0.83.43, and the write rate did not move with either. The DatumConnect v0.1.4 desktop release at 04:04Z on the 11th adds no client code over v0.1.3.

Which field churns. Corrected on 13 September, see below. The writes come from the NSO connector controller, not the agents. Agents patch only connection details, and only when those change, at a flat 0.003 a second. The connector reconcile that computes Ready from the agent's Lease writes only when status differs. Its reconciles that end with the lease not ready went from 0 an hour before 13:00Z to 100 to 370 an hour after, and in every three-hour window since they match the status update count within a few percent (222 vs 242, 611 vs 581, 711 vs 606, 612 vs 558, 801 vs 703, 401 vs 398, 780 vs 760, 668 vs 667, 490 vs 473). Lease renewals also roughly doubled at onset (about 1880 to 3300 to 3800 per three hours) with some Connector and Lease creates and deletes. A log sample of the replicator at 22:15Z on 12 September shows it handling two Connectors in one project (datum-connect-zr5z4 and datum-connect-bpb8q in project demos-md21mk) within four seconds.

This paragraph originally concluded that the churning field was the Ready condition flipping between True and False. A direct read of the objects on 13 September does not support that. Every Connector in demos-md21mk carries a Ready transition time of 11 September 18:12Z or earlier, apart from one on 13 September, and datum-connect-zr5z4 has not been written since about 15:10Z on 11 September. Across all 89 production Connectors no Ready transition falls inside 11 September 18:12Z to 13 September 00:45Z, which is most of the event. The correlation above still holds, but a reconcile that ends not ready and finds status unchanged writes nothing, so the two counts can track without the writes being Ready flips. The churning field is unidentified. Naming it needs audit logs for Connector status writes in the window, which is the visibility gap tracked in https://github.com/datum-cloud/infra/issues/5339.

Why every status change reaches the edge. The replicator copies the entire upstream status verbatim into the upstream-status annotation on the hub Connector (mirrorStatusToAnnotation, set for Connector in gateway_resource_replicator_controller.go, copy in setUpstreamStatusAnnotation). It skips writes when nothing changed, but any status change at all, including condition messages, lastTransitionTime, the IrohDNSPublished condition or relay addresses, changes the annotation. Karmada propagates the annotation, and the edge extension server reads only two things from it: whether Ready is True, and the public key id (connectorLiveness and connectorStatusLiveness in internal/extensionserver/cache/index.go). Most of the churn therefore buys a full rebuild that cannot change the programmed result, and the rest (true liveness flaps) triggers one rebuild per flap with no damping.

Sibling issue with the same shape on HTTPRoute: #438. Earlier Connector liveness fixes that this must not regress: #167 and #209.

Infra stopgap and its cost. Infra could drop Connector from the edge gateway controller's extension watch. That stops the rebuilds immediately, but a tunnel going online or offline would then reach the edge only on the next unrelated rebuild, which reintroduces the stale tunnel routing fixed in #167 and #209. We would rather fix it here.

# project control plane Connector writes by verb and subresource
sum by (verb, subresource) (increase(apiserver_request_total{job="milo-apiserver", resource="connectors", verb=~"PUT|PATCH|POST|DELETE"}[6h]))

# hub Connector writes (what Karmada propagates)
sum(increase(apiserver_request_total{job="karmada-system/karmada-apiserver", resource="connectors", verb="PUT"}[6h]))

# connector reconciles and status updates
sum(increase(controller_runtime_reconcile_total{cluster="prod-infrastructure-control-plane", controller="connector", result="success"}[3h]))
sum(increase(apiserver_request_total{job="milo-apiserver", resource="connectors", subresource="status", verb="PUT"}[3h]))

# agent lease renewals
sum(increase(apiserver_request_total{job="milo-apiserver", resource="leases", verb="PATCH"}[3h]))

# replicator reconciles
sum(increase(controller_runtime_reconcile_total{cluster="prod-infrastructure-control-plane", controller="gateway_resource_replicator"}[2m]))