feat(otelcollector): render OTel Collector

[tianfeng92]

Summary

Render the OTel Collector as a StatefulSet in calico-system, configured via LogCollector.spec.openTelemetry.

• Design doc
• Calico-private: #12834 builds the binary into the calico image, #13162 moves it to third_party/

The collector receives logs from fluent-bit via OTLP and optionally federates Prometheus metrics, forwarding both to user-configured OTLP endpoints. It is added to the LogCollector CR rather than AdditionalStores because it is operator-managed infrastructure (StatefulSet, ConfigMap, RBAC, certs) with its own lifecycle, not a pointer to an external system.

API

spec:
  openTelemetry:
    logs:
      types: [Flows, DNS, Audit]
    metrics:
      state: Enabled
    exporters:
      - name: datadog
        endpoint: https://otlp.datadoghq.com:4318
        protocol: http
        auth:
          headers:
            - name: DD-API-KEY
              valueFrom:
                secretKeyRef: { name: otlp-datadog-auth, key: api-key }
        tls:
          caConfigMapName: corp-ca
          clientCertSecretName: corp-client-mtls

TLS and auth are configured per exporter rather than once for all of them: this fans out to several external backends, which generally expect different client identities and different trust anchors. tls.clientCertSecretName turns on mutual TLS for that exporter alone; tls.caConfigMapName pins that exporter to the named CA alone — the system roots are not trusted as well, since keeping them would let any public authority satisfy the connection, which is what pinning is for. An exporter that names no CA verifies against the system roots. Verification is never disabled. Header credentials are read from Secrets into environment variables and referenced as ${env:...}, so they never appear in the rendered config.

Exporter CAs are deliberately kept out of the operator's trusted bundle — that bundle backs the receiver's client_ca_file, so folding them in would make them valid signers for inbound client certificates. The collector carries its own named bundle rather than the shared calico-system/tigera-ca-bundle, which the Installation controller owns.

Release Note

Add OpenTelemetry Collector component to export logs and metrics via OTLP.

Test plan

• Unit tests for render, controller, and the fluent-bit pipeline output
• Deployed to a cluster: all three log types reaching an OTLP backend, Prometheus federation exporting, collector's own ServiceMonitor scraping, clean startup and teardown
• Verified the collector is not deployed when the license lacks the feature, and that disabling export removes every resource it owns
• Full lifecycle on a cluster: enable → logs and metrics flowing → disable → all resources removed → re-enable → data resumes

Exercised on a cluster since: a pinned caConfigMapName renders ca_file with no system roots and is mounted and loaded, and a user-supplied otel-collector-tls reaches fluent-bit's trust bundle and is accepted with no verification errors.

Still unit-tested only: clientCertSecretName and auth.headers against a real TLS-terminating backend — no external OTLP endpoint demanding mutual TLS or a credential header was available to test against.

Follow-up: moving Dex and Guardian onto the shared ParseExternalDestination helper (r3717124352) is going into its own PR.

EV-6862

[pasanw]

Thanks for addressing my comments! I let Claude lead the re-review and reviewed everything it said was addressed along with what it flagged - it helped flag some things that weren't fully addressed but it also flagged some new issues (sorry about that!)

I'll let you and @rene-dekker work through them together and decide the best way to proceed.

[rene-dekker]

Ran Claude Code over this branch as a review pass; it flagged one thing that looks like it would stop the collector from starting at all, so raising it ahead of the smaller notes.

Exporter component IDs may not be valid otelcol types

pkg/render/otelcollector/component.go:76-77:

exporterPrefixHTTP  = "otlp_http"
exporterPrefixGRPC  = "otlp_grpc"

These are used as the component-ID type in two places — the exporters block ({{.Prefix}}/{{.Name}}:, collector-config.yaml.template:50) and every pipeline's exporters: [...] list (ExporterNames, component.go:511) — producing:

exporters:
  otlp_grpc/backend:
    endpoint: ...
service:
  pipelines:
    logs:
      exporters: [otlp_grpc/backend]

In an otelcol config the part before the / has to match a registered component factory type. Upstream those are otlp for the gRPC exporter and otlphttp for the HTTP one — otlp_grpc and otlp_http aren't factory types in any upstream distro. otelcol resolves the type at config-load time and a miss is fatal:

error decoding 'exporters': unknown type: "otlp_grpc" for id "otlp_grpc/backend"

If that applies here, the StatefulSet crash-loops on every install with an exporter configured, the health check never passes, and the TigeraStatus sits on Progressing with nothing pointing at the generated config as the cause.

The render tests (component_test.go:342, :382-404) assert these same prefixes, so they pass either way — they check that the operator renders what the operator intends, not that otelcol accepts it.

What I couldn't verify: the container runs /usr/bin/otelcol from the combined calico image, and the current tigera/calico:latest doesn't contain that binary yet (only calico, calico-ipam, calicoctl, csi-node-driver-registrar), so I couldn't run otelcol components against it. Two questions:

1. Does the companion build register custom factory types under these names? A factory's type comes from the factory itself, so otlp_grpc/otlp_http would need custom wrapper factories rather than a builder-manifest rename — if that's what's happening, worth a comment in the template saying so, because it reads as a bug.
2. If not, these want to be otlphttp and otlp.

Either way it'd be good to have one test that feeds a rendered config to the actual collector binary — the current tests can't distinguish the two cases.

[rene-dekker]

I believe we need a watch in this controller for the new secret otel-collector-tls

[tianfeng92]

Added to the secrets watch loop in add():

operator/pkg/controller/logcollector/logcollector_controller.go

Lines 136 to 138 in 83bfa6b

	// The collector's serving certificate goes into fluent-bit's trusted
	// bundle, so creating or rotating it has to rebuild that bundle.
	otelcollector.OpenTelemetryCollectorServerTLSSecretName,

and consumed in Reconcile, where the cert joins fluent-bit's trusted bundle when logs are exported:

operator/pkg/controller/logcollector/logcollector_controller.go

Lines 491 to 501 in 83bfa6b

	if instance.Spec.OpenTelemetry.HasLogs() {
	otelCertificate, err := certificateManager.GetCertificate(r.client, otelcollector.OpenTelemetryCollectorServerTLSSecretName, common.OperatorNamespace())
	if err != nil {
	r.status.SetDegraded(operatorv1.ResourceReadError, "Failed to get the OpenTelemetry Collector certificate", err, reqLogger)
	return reconcile.Result{}, err
	}
	// Absent simply means the otel controller has not minted it yet; it will
	// trigger another reconcile here when it does.
	if otelCertificate != nil {
	extraCerts = append(extraCerts, otelCertificate)
	}

[rene-dekker]

Another one from the same Claude Code review pass, much smaller than the last:

Logs.Types accepts duplicates, which render duplicate fluent-bit outputs

api/v1/opentelemetry_types.go:34-38 declares the field with an item-level enum but no uniqueness constraint:

// +optional
Types []OpenTelemetryLogType `json:"types,omitempty"`

The generated CRD (operator.tigera.io_logcollectors.yaml:1110-1123) is a plain type: array with an item enum and no x-kubernetes-list-type: set, so the API server accepts types: [Flows, Flows]. Nothing downstream dedupes it either — otelLogTypes() (logcollector_controller.go:952-957) passes the slice straight through, and pipeline.go:255 iterates it appending one output per entry:

outputs:
  - name: opentelemetry
    match: flows
    ...
  - name: opentelemetry
    match: flows
    ...

fluent-bit runs both instances, so every flow record is shipped to the collector — and on to the customer's OTLP backend — twice. For a usage-priced backend that is a duplicated bill as well as duplicated data. With NonClusterHost configured it's four outputs, since each type already renders a second output for the non-cluster tag.

Fix is a marker:

// +optional
// +listType=set
Types []OpenTelemetryLogType `json:"types,omitempty"`

Exporters already has +listType=map + +listMapKey=name for the same reason, so this looks like an oversight rather than a decision. Note Validate() guards against duplicate exporter names in Go as well, with a comment about the CRD being out of date — if that concern applies here too, the same belt-and-braces dedupe would be worth adding for log types.

[caseydavenport]

Most of these comments are just me scolding Claude for being way too verbose in its code comments....

[caseydavenport]

A bit of a high level question here, but is LogCollector really the right place for this?

We're exporting more than just logs ( e.g., prometheus metrics ) - so I wonder if this really wants its own CRD?

[caseydavenport]

nit, but I think we have a new enterprise subdir in a few places for utilities - we should try to keep any utils we're adding within those directories where appropriate (this is sort of more general for the whole PR).

[caseydavenport]

AI overcommenting here.

[caseydavenport]

This massive comment block also looks like AI over-commenting to me.

I think this should be as simple as:

// FluentBit needs system certificates to talk to external tools like Splunk, Syslog, or AWS. We give 
// it its own trusted bundle as a result to avoid contamination with other services in this namespace. 
//
// We must also trust the OTEL collector's server certificate, if enabled.

[caseydavenport]

Suggested change

	// Same predicate the otel controller deploys on. Pointing fluent-bit at a
	// collector that is never rendered — unlicensed, or an invalid spec — just
	// fails every chunk and fills the storage buffer.
	OpenTelemetryCollectorEnabled: instance.Spec.OpenTelemetry.Deployable(
	utils.IsFeatureActive(license, common.OpenTelemetryCollectorFeature)),
	OpenTelemetryCollectorEnabled: instance.Spec.OpenTelemetry.Deployable(utils.IsFeatureActive(license, common.OpenTelemetryCollectorFeature)),

[caseydavenport]

I think this comment is wrong / out of date? Second return is a map of Secrets.

[caseydavenport]

Hm, is there not an upstream regex we could import for this?

[caseydavenport]

Again, self-explanatory from the code, Claude.

Suggested change

	// OpenTelemetryEnabled reports whether the OTel Collector is configured. Its
	// ServiceMonitor is created and removed with the collector; without this the
	// monitor render cannot tell, and would leave one behind selecting a Service
	// that no longer exists.

[caseydavenport]

Suggested change

	// Nothing is owned by a CR we can rely on being deleted, so switching the
	// Objects are owned by a sub-struct of the LogCollector CR rather than the CR as a whole, so switching the

[caseydavenport]

Hm, this is sketchy - we have a policy of disallowing HTTP in our product. I don't think we should allow this - it's a bad security posture that we are enabling.