Skip to content

fix(alerts): reduce reconcile storm flapping - #362

Merged
scotwells merged 1 commit into
mainfrom
fix/reconcile-storm-alert-flapping
Sep 24, 2026
Merged

scotwells merged 1 commit into
mainfrom
fix/reconcile-storm-alert-flapping

Conversation

@scotwells

Copy link
Copy Markdown
Contributor

What this changes

Reduce alert churn from ComputeControllerReconcileStorm:

  • use the five-minute average workqueue depth instead of the maximum, so a brief depth spike does not immediately resolve a sustained storm;
  • remove ComputeControllerWorkqueueAddStorm, which pages on the same underlying condition as the reconcile-rate alert;
  • keep workqueue-add rate in the runbook as corroborating diagnostic evidence.

The primary alert still requires a sustained high reconcile rate and a mostly drained queue, so a genuinely busy controller with a persistent backlog remains suppressed.

Validation

  • kustomize build config/overlays/management-plane
  • kustomize build config/overlays/single-cluster

Related: #355 and service-catalog#107.

@scotwells
scotwells requested a review from a team September 24, 2026 16:02
@scotwells
scotwells enabled auto-merge September 24, 2026 16:05
@scotwells
scotwells merged commit 571f1f3 into main Sep 24, 2026
11 checks passed
@scotwells
scotwells deleted the fix/reconcile-storm-alert-flapping branch September 24, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants