Skip to content

CP-44601: Aggregate CloudZeroWebhookNoEvents with sum() to stop per-series misfires #906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
evan-cz wants to merge 1 commit into
base: develop
Choose a base branch
from
+26 −10
Open

CP-44601: Aggregate CloudZeroWebhookNoEvents with sum() to stop per-series misfires #906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions helm/docs/monitoring-infrastructure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,15 +99,15 @@ organized into three categories.

These indicate the agent is broken and not collecting or delivering data.

| Alert | Metric | For | Condition | What It Catches |
| -------------------------------- | -------------------------------------------------- | ---------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CloudZeroAgentDown` | `up` / `absent(up)` | 5m | target down or absent | Prometheus agent pod unreachable or not running. No cluster metrics are being collected. Uses `absent()` to detect pods that are unschedulable or completely missing, not just unhealthy. |
| `CloudZeroWebhookDown` | `up` / `absent(up)` | 5m | target down or absent | Webhook pod unreachable or not running. Kubernetes resource metadata not being captured. |
| `CloudZeroCollectorNoMetrics` | `metrics_received_total` | 15m | `rate == 0` | Collector is running but not receiving remote-write data from the Prometheus agent. |
| `CloudZeroAggregatorDown` | `up` / `absent(up)` | 5m | target down or absent | Aggregator pod (collector + shipper) unreachable or not running. No metrics are being collected or uploaded. |
| `CloudZeroShipperUploadFailures` | `shipper_run_fail_total` | 3.1x `costMaxInterval` | `rate > 0` | Shipper is failing to upload files. Threshold is dynamic: 3.1x the configured `aggregator.database.costMaxInterval` (default 30m = 93m). The `error_status_code` label indicates the failure category (`err-unauthorized`, `err-network`, etc.). |
| `CloudZeroWebhookNoEvents` | `czo_webhook_types_total` | 30m | `rate == 0` | Webhook is running but receiving zero admission requests. Most dangerous silent failure -- typically indicates a TLS certificate mismatch after cert rotation. |
| `CloudZeroAgentOOMKilled` | `kube_pod_container_status_last_terminated_reason` | 1m | `reason="OOMKilled"` | A container was killed by the kernel OOM killer. The memory limit is too low for the workload. |
| Alert | Metric | For | Condition | What It Catches |
| -------------------------------- | -------------------------------------------------- | ---------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CloudZeroAgentDown` | `up` / `absent(up)` | 5m | target down or absent | Prometheus agent pod unreachable or not running. No cluster metrics are being collected. Uses `absent()` to detect pods that are unschedulable or completely missing, not just unhealthy. |
| `CloudZeroWebhookDown` | `up` / `absent(up)` | 5m | target down or absent | Webhook pod unreachable or not running. Kubernetes resource metadata not being captured. |
| `CloudZeroCollectorNoMetrics` | `metrics_received_total` | 15m | `rate == 0` | Collector is running but not receiving remote-write data from the Prometheus agent. |
| `CloudZeroAggregatorDown` | `up` / `absent(up)` | 5m | target down or absent | Aggregator pod (collector + shipper) unreachable or not running. No metrics are being collected or uploaded. |
| `CloudZeroShipperUploadFailures` | `shipper_run_fail_total` | 3.1x `costMaxInterval` | `rate > 0` | Shipper is failing to upload files. Threshold is dynamic: 3.1x the configured `aggregator.database.costMaxInterval` (default 30m = 93m). The `error_status_code` label indicates the failure category (`err-unauthorized`, `err-network`, etc.). |
| `CloudZeroWebhookNoEvents` | `czo_webhook_types_total` | 30m | `sum(rate) == 0` | Webhook is running but receiving zero admission requests fleet-wide. Most dangerous silent failure -- typically a TLS certificate mismatch after cert rotation. Aggregated with `sum()` + `or vector(0)` so it reflects the whole fleet, not a single quiet series. |
| `CloudZeroAgentOOMKilled` | `kube_pod_container_status_last_terminated_reason` | 1m | `reason="OOMKilled"` | A container was killed by the kernel OOM killer. The memory limit is too low for the workload. |

### Warning Alerts

Expand Down
18 changes: 17 additions & 1 deletion helm/templates/prometheusrule.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,11 +151,27 @@ spec:
it typically indicates a TLS certificate mismatch after rotation.

Uses czo_webhook_types_total from app/domain/webhook/webhook.go.

The expression MUST aggregate with sum(). czo_webhook_types_total is a
counter vector split by kind_group/kind_version/kind_resource/operation
(and per pod/instance by the scrape), so a bare "rate(...) == 0" is
evaluated per series and fires whenever ANY single series is flat --
e.g. a rarely-seen resource/operation (storageclass creates) or a quiet
replica -- even while the webhook is healthy and busy overall. sum()
collapses those into a single fleet-wide rate so the alert only fires
when NO replica sees ANY admission event of ANY type.

The "or vector(0)" is also required: the counter has no series at all
until the first admission event is processed, so on a webhook that has
received nothing since startup (the exact failure this catches) the
metric is absent and a bare aggregate would return an empty vector and
never fire. "or vector(0)" substitutes 0 in that case so the alert
fires on genuine silence.
See: Monitoring the CloudZero Agent > Monitoring Webhook TLS Certificate Health
*/}}
{{- if eq (include "cloudzero-agent.webhookServer.enabled" .) "true" }}
- alert: CloudZeroWebhookNoEvents
expr: rate(czo_webhook_types_total{job="{{ include "cloudzero-agent.serviceName" . }}"}[30m]) == 0
expr: (sum(rate(czo_webhook_types_total{job="{{ include "cloudzero-agent.serviceName" . }}"}[30m])) or vector(0)) == 0
for: 30m
labels:
severity: critical
Expand Down
Loading