EV-6666: Surface Alertmanager alerts on the manager Alerts page - #4879
Draft
rene-dekker wants to merge 16 commits into
Draft
EV-6666: Surface Alertmanager alerts on the manager Alerts page#4879rene-dekker wants to merge 16 commits into
rene-dekker wants to merge 16 commits into
Conversation
|
|
rene-dekker
force-pushed
the
EV-6666
branch
4 times, most recently
from
June 10, 2026 20:22
044b64c to
249f607
Compare
electricjesus
left a comment
electricjesus
left a comment
Member
There was a problem hiding this comment.
Drive-by review, courtesy of a quest from Tigera Town 🤣. Mostly looks good. One thing I think blocks merge, plus a few mediums, left inline.
Cross-PR ordering: these two have to ship together, and this PR can't stand alone. The operator's own RBAC for alertmanagerconfigs lives in the calico-private charts in tigera/calico-private#12184, so if this vendors into the operator ahead of that, the monitor controller can't create the AlertmanagerConfig and goes degraded. Same on the receiving end: without #12184 the /api/v1/events/alertmanager endpoint 404s and Linseed rejects the prometheus_alert type. Worth pinning both to the same release and noting the dependency on each PR while they're still draft.
One nice-to-have I noticed but won't block on: the config-hash annotation that rolls the pod doesn't include the token secret data, so the pod won't roll when Kubernetes first populates the token. It relies on the config-reloader watching the mounted secret. Probably fine, worth a sanity check.
Add a Linseed network policy ingress rule permitting traffic from the Alertmanager pods in the tigera-prometheus namespace, so Alertmanager can push Prometheus alerts to Linseed as events. The Alertmanager egress policy already allows all TCP egress, so only the Linseed ingress side was missing. Exports monitor.AlertmanagerSourceEntityRule as the single source of truth for the Alertmanager pod selector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a ClusterRole granting create on events (linseed.tigera.io), bound to the prometheus service account that Alertmanager runs as. Linseed authorizes writes via SubjectAccessReview, so this lets Alertmanager push Prometheus alerts to Linseed as events using its existing service account token. The role/binding are rendered only when Alertmanager is enabled and removed otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the placeholder Alertmanager webhook receiver with one that posts to Linseed's /api/v1/events/alertmanager endpoint, so Prometheus alerts surface on the Alerts UI page. Linseed requires mTLS plus a bearer token, so the Alertmanager spec now mounts the prometheus client TLS key pair and the trusted CA bundle, and the webhook http_config references them along with the service account token. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a UIAlertsIntegration (Enabled|Disabled) field to the Monitor spec that controls whether Prometheus/Alertmanager alerts are forwarded to Linseed and surfaced on the manager Alerts page (defaults to Enabled). When disabled, the operator renders an Alertmanager config that routes to a null receiver instead of the Linseed webhook. The config secret is regenerated to the selected variant when the operator owns it, so the toggle takes effect at runtime. A hash of the Alertmanager config is added as a pod annotation so that config changes roll the Alertmanager pod and reload the new config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the raw alertmanager.yaml config secret with an AlertmanagerConfig custom resource referenced by Alertmanager.spec.alertmanagerConfiguration: - If the user supplies an AlertmanagerConfig named calico-node-alertmanager in the tigera-operator namespace, the operator renders a copy of it in tigera-prometheus. Otherwise it renders a default: the Linseed webhook receiver when the UI alerts integration is enabled, or a null receiver when disabled. - The webhook authenticates to Linseed with the Linseed-issued bearer token secret for the prometheus service account (prometheus-tigera-linseed-token) and the client cert / CA bundle, all referenced from the CR; the prometheus-operator mounts them into the Alertmanager pod, so the explicit Secrets/ConfigMaps mounts are removed. - The pod is annotated with a hash of the AlertmanagerConfig spec, client cert and CA bundle so any config change rolls the pod. - The legacy alertmanager-calico-node-alertmanager config secret is now deleted. This also fixes the upgrade gap where a pre-existing (stock) config secret was left untouched because it matched neither operator default, so the integration never wired up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 044b64c)
Add IPPoolNearlyExhausted (>=90%, warning) and IPPoolExhausted (>=100%, critical) rules to the rendered calico PrometheusRule. Utilisation is computed per pool as sum by (ippool) (ipam_allocations_in_use) / sum by (ippool) (ipam_ippool_size); summing both sides on ippool aggregates the per-node allocations and collapses scrape labels so the metrics match on ippool alone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the single UIAlertsIntegration toggle with a per-alert config under
Monitor.spec.alerts. The alert set is a curated, closed catalog (one field per
alert), each {Status: Enabled|Disabled}, defaulting to Enabled when unset so new
alerts ship on automatically. Warning+critical rule pairs are folded into one
logical alert. Fields: deniedPackets, tigeraStatus, tlsCertExpiry, licenseExpiry,
ipPoolExhaustion.
Also rename the DeniedPacketsRate rule to DeniedPackets (drop "high rate" wording)
and default Alertmanager to 1 replica so the alerts feature is on by default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…account Give Alertmanager its own service account (calico-alertmanager) instead of reusing the shared prometheus SA: create the SA, run Alertmanager under it, mint its Linseed bearer-token secret for it, and bind the events ClusterRole to it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oken RBAC Revert AlertmanagerConfig-CR delivery to a raw alertmanager.yaml config Secret, folding Monitor.spec.alerts enable/disable into routing (enabled alerts -> the linseed webhook, everything else -> a null receiver). Drops the operator's AlertmanagerConfig watch (and the need for alertmanagerconfigs RBAC). On managed clusters, also grant the management cluster's guardian service account permission to manage secrets in tigera-prometheus (the tigera-linseed RoleBinding), so Linseed's token controller can provision the calico-alertmanager token there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…clusters
On a managed cluster the Alertmanager Linseed token is a Linseed-issued JWT that
Linseed's token controller pushes into tigera-prometheus as an Opaque secret. The
operator was also creating a service-account-token secret of the same name, and
since a Secret's type is immutable the two collide ("type Opaque is immutable"),
so the managed-cluster token never provisions. Gate the operator's SA-token secret
on !ManagedCluster (and delete it on managed clusters) so the token controller owns it there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a managed cluster the Alertmanager webhook could not reach Linseed, so UI alerts never made it to the management cluster. Three gaps: - The webhook URL/SNI hardcoded the in-cluster service (tigera-linseed.tigera-elasticsearch.svc), which does not resolve on a managed cluster. Address Linseed via a namespace-local "tigera-linseed" ExternalName service that redirects to Guardian (SNI "tigera-linseed"), mirroring fluentd. - The operator deleted the Alertmanager Linseed token Secret on managed clusters, wiping the Linseed-issued JWT that the token controller owns (same name, different immutable type). The operator now neither creates nor deletes that Secret on managed clusters. - Alertmanager's trusted bundle lacked the management cluster's Linseed CA (the webhook TLS connection terminates at the management Linseed through the tunnel). Add VoltronLinseedPublicCert to the bundle on managed clusters, mirroring fluentd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… clusters The monitor controller read the existing Alertmanager Linseed token secret on every reconcile to carry its Kubernetes-populated data forward. On a managed cluster the operator does not render that secret (Linseed's token controller owns it as an Opaque JWT), so the read was pointless and, being type-blind, could copy Linseed's JWT into the operator's desired secret. Gate the read on !managedCluster so the controller only reads it where the renderer creates it. Also make MonitorSpec.UIAlertsEnabled use a pointer receiver to match the generated DeepCopy methods (staticcheck ST1016), and trim verbose doc comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the per-alert Monitor.spec.alerts API (Alerts/Alert/AlertStatusType and UIAlertsEnabled). Alertmanager now routes every alert to the Linseed events webhook unconditionally, so all built-in alerts surface on the Manager Alerts page. Users suppress individual alerts via AlertExceptions rather than through the Monitor API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shorten the Alertmanager rendering comments and drop in-code cross-references per the comment style. Also fix two stale docs that named the Prometheus service account where the Linseed role now binds the dedicated calico-alertmanager one, and remove the unused AlertmanagerConfigName const left over from the switch to raw alertmanager.yaml delivery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rene-dekker
commented
Jul 17, 2026
| type AlertmanagerSpec struct { | ||
| // Replicas defines the number of Alertmanager replicas. When set to 0, Alertmanager is not rendered. | ||
| // Default: 0 | ||
| // Default: 1 |
Member
Author
There was a problem hiding this comment.
note: the default of 0 was never released, so this is not a change.
Remove references to an AlertmanagerConfig CR and the Monitor alerts API that only make sense against approaches that never shipped, and clarify that the Linseed token carry-forward is the non-managed path (Linseed's token controller owns that secret on managed clusters). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove doc comments that restate the identifier, section labels, and plumbing narration. Keep only the non-obvious constraints: the immutable service-account- token vs Linseed-JWT distinction, the per-ippool PromQL aggregation, and the non-managed token carry-forward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires Prometheus/Alertmanager alerts through to the manager Alerts page.
calico-alertmanagerservice account; issue it a Linseed bearer token (a service-account-token Secret, validated via TokenReview) plus RBAC to create Linseed events.alertmanager.yamlsecret whose webhook forwards all alerts to Linseed's/api/v1/events/alertmanagerover mTLS. Users suppress individual alerts via AlertExceptions, not the Monitor API.tigera-linseedExternalName service; the token is provisioned by Linseed's token controller.Companion PRs: calico-private (Linseed ingest + dedup), ui-modules (
prometheus_alertrendering).🤖 Generated with Claude Code