From 30b8dd6632c325025dffa2b3f542d588c5e051f6 Mon Sep 17 00:00:00 2001 From: Evan Nemerson Date: Tue, 14 Jul 2026 09:47:19 -0400 Subject: [PATCH] CP-44601: Aggregate CloudZeroWebhookNoEvents with sum() to stop per-series misfires The CloudZeroWebhookNoEvents alert used a bare per-series expression (rate(czo_webhook_types_total[30m]) == 0) that fired whenever any single series was flat -- a rarely-seen resource/operation, or one quiet replica -- even while the webhook was healthy and busy overall. It also never fired on a webhook that had received nothing since startup, because the counter has no series until the first admission event is processed. Aggregate fleet-wide with sum(rate(...)) and add "or vector(0)" so genuine silence (an absent metric) still fires. Update the template comment and the monitoring-infrastructure alert reference to match. Verified: helm template renders the corrected expression; helm-unittest webhookserver_gating_test.yaml passes 36/36. Co-Authored-By: Claude opus-4.8 --- helm/docs/monitoring-infrastructure.md | 18 +++++++++--------- helm/templates/prometheusrule.yaml | 18 +++++++++++++++++- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/helm/docs/monitoring-infrastructure.md b/helm/docs/monitoring-infrastructure.md index 686f492d..67887b2a 100644 --- a/helm/docs/monitoring-infrastructure.md +++ b/helm/docs/monitoring-infrastructure.md @@ -99,15 +99,15 @@ organized into three categories. These indicate the agent is broken and not collecting or delivering data. -| Alert | Metric | For | Condition | What It Catches | -| -------------------------------- | -------------------------------------------------- | ---------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `CloudZeroAgentDown` | `up` / `absent(up)` | 5m | target down or absent | Prometheus agent pod unreachable or not running. No cluster metrics are being collected. Uses `absent()` to detect pods that are unschedulable or completely missing, not just unhealthy. | -| `CloudZeroWebhookDown` | `up` / `absent(up)` | 5m | target down or absent | Webhook pod unreachable or not running. Kubernetes resource metadata not being captured. | -| `CloudZeroCollectorNoMetrics` | `metrics_received_total` | 15m | `rate == 0` | Collector is running but not receiving remote-write data from the Prometheus agent. | -| `CloudZeroAggregatorDown` | `up` / `absent(up)` | 5m | target down or absent | Aggregator pod (collector + shipper) unreachable or not running. No metrics are being collected or uploaded. | -| `CloudZeroShipperUploadFailures` | `shipper_run_fail_total` | 3.1x `costMaxInterval` | `rate > 0` | Shipper is failing to upload files. Threshold is dynamic: 3.1x the configured `aggregator.database.costMaxInterval` (default 30m = 93m). The `error_status_code` label indicates the failure category (`err-unauthorized`, `err-network`, etc.). | -| `CloudZeroWebhookNoEvents` | `czo_webhook_types_total` | 30m | `rate == 0` | Webhook is running but receiving zero admission requests. Most dangerous silent failure -- typically indicates a TLS certificate mismatch after cert rotation. | -| `CloudZeroAgentOOMKilled` | `kube_pod_container_status_last_terminated_reason` | 1m | `reason="OOMKilled"` | A container was killed by the kernel OOM killer. The memory limit is too low for the workload. | +| Alert | Metric | For | Condition | What It Catches | +| -------------------------------- | -------------------------------------------------- | ---------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CloudZeroAgentDown` | `up` / `absent(up)` | 5m | target down or absent | Prometheus agent pod unreachable or not running. No cluster metrics are being collected. Uses `absent()` to detect pods that are unschedulable or completely missing, not just unhealthy. | +| `CloudZeroWebhookDown` | `up` / `absent(up)` | 5m | target down or absent | Webhook pod unreachable or not running. Kubernetes resource metadata not being captured. | +| `CloudZeroCollectorNoMetrics` | `metrics_received_total` | 15m | `rate == 0` | Collector is running but not receiving remote-write data from the Prometheus agent. | +| `CloudZeroAggregatorDown` | `up` / `absent(up)` | 5m | target down or absent | Aggregator pod (collector + shipper) unreachable or not running. No metrics are being collected or uploaded. | +| `CloudZeroShipperUploadFailures` | `shipper_run_fail_total` | 3.1x `costMaxInterval` | `rate > 0` | Shipper is failing to upload files. Threshold is dynamic: 3.1x the configured `aggregator.database.costMaxInterval` (default 30m = 93m). The `error_status_code` label indicates the failure category (`err-unauthorized`, `err-network`, etc.). | +| `CloudZeroWebhookNoEvents` | `czo_webhook_types_total` | 30m | `sum(rate) == 0` | Webhook is running but receiving zero admission requests fleet-wide. Most dangerous silent failure -- typically a TLS certificate mismatch after cert rotation. Aggregated with `sum()` + `or vector(0)` so it reflects the whole fleet, not a single quiet series. | +| `CloudZeroAgentOOMKilled` | `kube_pod_container_status_last_terminated_reason` | 1m | `reason="OOMKilled"` | A container was killed by the kernel OOM killer. The memory limit is too low for the workload. | ### Warning Alerts diff --git a/helm/templates/prometheusrule.yaml b/helm/templates/prometheusrule.yaml index a1b99fb8..0cf94b3a 100644 --- a/helm/templates/prometheusrule.yaml +++ b/helm/templates/prometheusrule.yaml @@ -151,11 +151,27 @@ spec: it typically indicates a TLS certificate mismatch after rotation. Uses czo_webhook_types_total from app/domain/webhook/webhook.go. + + The expression MUST aggregate with sum(). czo_webhook_types_total is a + counter vector split by kind_group/kind_version/kind_resource/operation + (and per pod/instance by the scrape), so a bare "rate(...) == 0" is + evaluated per series and fires whenever ANY single series is flat -- + e.g. a rarely-seen resource/operation (storageclass creates) or a quiet + replica -- even while the webhook is healthy and busy overall. sum() + collapses those into a single fleet-wide rate so the alert only fires + when NO replica sees ANY admission event of ANY type. + + The "or vector(0)" is also required: the counter has no series at all + until the first admission event is processed, so on a webhook that has + received nothing since startup (the exact failure this catches) the + metric is absent and a bare aggregate would return an empty vector and + never fire. "or vector(0)" substitutes 0 in that case so the alert + fires on genuine silence. See: Monitoring the CloudZero Agent > Monitoring Webhook TLS Certificate Health */}} {{- if eq (include "cloudzero-agent.webhookServer.enabled" .) "true" }} - alert: CloudZeroWebhookNoEvents - expr: rate(czo_webhook_types_total{job="{{ include "cloudzero-agent.serviceName" . }}"}[30m]) == 0 + expr: (sum(rate(czo_webhook_types_total{job="{{ include "cloudzero-agent.serviceName" . }}"}[30m])) or vector(0)) == 0 for: 30m labels: severity: critical