Deregister control plane VM from Kubernetes_API_Server LB rule on CloudStackMachine delete

[metbog]

Issue #, if available:

N/A

Description of changes:

Currently, when a control plane CloudStackMachine is deleted, CAPC only destroys the underlying CloudStack VM - it never detaches the VM from the Kubernetes_API_Server load balancer rule. VM removal can take a while, and the CloudStack LB health check only marks the member offline after some delay, so API traffic keeps hitting the about-to-be-expunged kube-apiserver throughout that window. This causes stale members and intermittent control plane reachability issues during scale-down or rolling replace (most visibly during KCP-driven control plane rollouts)

This change mirrors the CAPO (cluster-api-provider-openstack) DeleteLoadBalancerMember pattern, with one addition: removal is attempted early - as soon as the parent CAPI Machine is marked for deletion - instead of only on the CloudStackMachine delete path. The late removal is kept as a safety net

Changes

• pkg/cloud/isolated_network.go - added RemoveVMFromLoadBalancerRule to the IsoNetworkIface. Idempotent: returns nil when LBRuleID is empty or the VM is not currently a member of the rule. Otherwise calls CloudStack removeFromLoadBalancerRule

• controllers/cloudstackmachine_controller.go - two LB-detach stages plus a small helper:

  • usesAPIServerLB() - extracted predicate shared by all LB stages: non-routed isolated network + populated IsoNet + InstanceID set
  • RemoveFromLBIfMachineDeleting - new stage in the regular Reconcile() pipeline, ordered before AddToLBIfNeeded. Fires when the parent CAPI Machine has a DeletionTimestamp, the machine is a control plane, and usesAPIServerLB() is true. Detaches the VM from the LB rule promptly, before KCP starts cordon/drain. AddToLBIfNeeded has a matching guard so a deleting Machine never gets re-added in the same loop
  • RemoveFromLBIfNeeded - new stage called from ReconcileDelete immediately before DestroyVMInstance, acting as a safety net for the case where the early step did not run (CSMachine deleted directly, controller downtime, etc.). Two implementation notes:
    • Detects control plane via the clusterv1.MachineControlPlaneLabel on the CloudStackMachine itself, because on the delete path the common stages skip loading the CAPI parent Machine (RunIf(... DeletionTimestamp().IsZero() ..., GetParent(...))).
    • Fetches the CloudStackIsolatedNetwork via GetObjectByName(IsoNetMetaName(...), r.IsoNet), since the delete path does not pre-load it (unlike the regular Reconcile path)
  • CAPI Machine watch predicate widened - the existing clusterv1.Machine update predicate only enqueued the owned CloudStackMachine when Bootstrap.DataSecretName went from nil to set. It now also enqueues when DeletionTimestamp transitions from zero to non-zero. Without this, the early-removal stage would only run on the next periodic resync (1m default) - too late.

• pkg/cloud/isolated_network_test.go - 5 new unit tests covering: empty LBRuleID no-op, VM not a member no-op, happy-path removal, list error, remove error

• pkg/mocks/mock_client.go - regenerated via make generate-mocks (gitignored; auto-rebuilt by make test)

• docs/book/src/topics/cloudstack-permissions.md - added the required removeFromLoadBalancerRule permission

Behavior matrix

Trigger	Machine role	FD network	RoutingMode	Result
Parent CAPI Machine DeletionTimestamp set	Control plane	Isolated	"" (non-routed)	Early LB detach during regular Reconcile(), before drain
CloudStackMachine deletion (ReconcileDelete)	Control plane	Isolated	"" (non-routed)	Safety-net LB detach, then VM destroyed
Either trigger	Control plane	Isolated	non-empty (routed)	No LB call (no LB used)
Either trigger	Control plane	Shared / other	-	No LB call
Either trigger	Worker	any	-	No LB call
Either trigger	Any	IsoNet CR already gone / LBRuleID empty	-	Cloud-layer no-op (idempotent)
Either trigger	Any	VM not a member of the rule	-	Cloud-layer no-op (idempotent)

The cluster-scoped LB rule itself continues to be cleaned up through DisposeIsoNetResources on cluster teardown (unchanged) - only members are removed on machine delete, matching the CAPO model

Trade-offs / why both stages

The early stage (RemoveFromLBIfMachineDeleting) is what actually shortens the bad window during KCP rollouts. The late stage (RemoveFromLBIfNeeded in ReconcileDelete) is intentionally kept because:

• The controller may have been down when the parent Machine was marked for deletion
• A user can delete the CloudStackMachine directly, bypassing the CAPI Machine
• It costs at most one extra ListLoadBalancerRuleInstances call (idempotent no-op when already detached)

Testing performed:

• go build ./... - clean
• go vet ./controllers/... ./pkg/cloud/... - clean
• REPO_ROOT=$(pwd) go run github.com/onsi/ginkgo/v2/ginkgo --label-filter='!integ' ./pkg/cloud - 137 passed, 0 failed, 24 integration specs skipped as expected. All 5 new specs green
• Manually verified the regenerated mock surface includes RemoveVMFromLoadBalancerRule on MockClient

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-cloudstack ready!

Name	Link
🔨 Latest commit	c7c912b
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-cluster-api-cloudstack/deploys/6a0f290ead5f740008e7eb13
😎 Deploy Preview	https://deploy-preview-485--kubernetes-sigs-cluster-api-cloudstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: metbog
Once this PR has been reviewed and has the lgtm label, please assign dims for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @metbog!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-cloudstack 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-cloudstack has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: metbog / name: Metbog (c7c912b)

[k8s-ci-robot]

Hi @metbog. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.