CP-44601: Aggregate CloudZeroWebhookNoEvents with sum() to stop per-series misfires

[evan-cz]

Problem

The CloudZeroWebhookNoEvents PrometheusRule alert fires on healthy webhooks. Its expression

rate(czo_webhook_types_total{job="..."}[30m]) == 0

is evaluated per series. czo_webhook_types_total is a counter split by kind_group/kind_version/kind_resource/operation (and per pod/instance by the scrape), so the alert fires whenever any single series is flat for 30m — a rarely-seen resource/operation (e.g. storageclass creates) or one quiet replica — even while the webhook is healthy and busy overall.

It also never fires in the case it is meant to catch: the counter has no series until the first admission event, so a webhook that is up but receiving nothing produces an absent metric and an empty result.

Fix

(sum(rate(czo_webhook_types_total{job="..."}[30m])) or vector(0)) == 0

• sum() collapses the per-label and per-pod series into a single fleet-wide rate, so the alert fires only when no replica sees any admission event of any type.
• or vector(0) substitutes 0 when the metric is absent, so genuine silence still fires.

The template comment and helm/docs/monitoring-infrastructure.md are updated to match.

Verification

• helm template renders the corrected expression.
• helm-unittest webhookserver_gating_test.yaml passes 36/36.

This PR was created with AI assistance (Claude Code by Anthropic).

[greptile-apps]

Greptile Summary

This PR fixes the CloudZeroWebhookNoEvents PrometheusRule alert, which was misfiring on healthy webhooks because the expression was evaluated per label-combination series rather than across the fleet as a whole.

• Alert expression: changed from rate(czo_webhook_types_total{job=\"...\"}[30m]) == 0 (per-series, fires on any single quiet label/replica) to (sum(rate(czo_webhook_types_total{job=\"...\"}[30m])) or vector(0)) == 0 — sum() collapses all series into one fleet-wide rate, and or vector(0) ensures the alert still fires when the metric is entirely absent (no admission events since pod startup).
• Comment block: expanded with a clear explanation of why both sum() and or vector(0) are required, which is valuable for future maintainers.
• Documentation: helm/docs/monitoring-infrastructure.md table updated to reflect the new condition and behavior.

Confidence Score: 5/5

Safe to merge — this is a targeted, well-reasoned fix to an alerting rule with no impact on data collection or delivery paths.

The PromQL expression change is correct: sum() without a by() clause collapses all per-series data into a single fleet-wide instant vector, and or vector(0) correctly substitutes 0 when the metric is absent rather than returning an empty vector. Label stripping from sum() is intentional here — this is a fleet-wide signal, so a single unlabeled alert instance is exactly right. The for: 30m and [30m] window combination is unchanged and well-understood. The comment block added to the template is unusually thorough and will prevent future regressions.

No files require special attention.

Important Files Changed

Filename	Overview
helm/templates/prometheusrule.yaml	Alert expression updated from per-series rate(...) == 0 to (sum(rate(...)) or vector(0)) == 0; well-commented with detailed rationale for both the aggregation and the absent-metric fallback.
helm/docs/monitoring-infrastructure.md	Documentation table updated to reflect the new sum(rate) == 0 condition and expanded description for CloudZeroWebhookNoEvents; purely a doc-sync change, no logic here.

_{Reviews (1): Last reviewed commit: "CP-44601: Aggregate CloudZeroWebhookNoEv..." | Re-trigger Greptile}