Add host metadata challenges demo: multi-pipeline OTel setup - #213
Open
IbraheemA wants to merge 1 commit into
Open
Add host metadata challenges demo: multi-pipeline OTel setup#213IbraheemA wants to merge 1 commit into
IbraheemA wants to merge 1 commit into
Conversation
Demonstrates how a standard Kubernetes OTel deployment (DaemonSet agent + Deployment gateway) produces three independent OTLP payload shapes — each carrying a different slice of host/workload identity: 1. App traces with k8s.* workload attributes (from k8sattributes processor) 2. Host metrics with host.* attributes (from hostmetrics receiver) 3. kube-state-metrics with Kubernetes object-state labels (from Prometheus scrape) Includes a Go trace-generator app, agent/gateway collector configs, kube-state-metrics deployment, Makefile for kind cluster setup, and a detailed README explaining exactly how and why the separation occurs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mx-psi
reviewed
Apr 10, 2026
| @@ -0,0 +1,11 @@ | |||
| FROM golang:1.22-alpine AS builder | |||
Member
There was a problem hiding this comment.
I think if we merge this we will immediately get a PR to update this Go version so we may as well upgrade it here 😄
Comment on lines
+21
to
+29
| endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") | ||
| if endpoint == "" { | ||
| endpoint = "localhost:4317" | ||
| } | ||
|
|
||
| exporter, err := otlptracegrpc.New(context.Background(), | ||
| otlptracegrpc.WithEndpoint(endpoint), | ||
| otlptracegrpc.WithInsecure(), | ||
| ) |
Member
There was a problem hiding this comment.
I think oteltracegrpc already supports this env variable, see https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc#pkg-overview
Suggested change
| endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") | |
| if endpoint == "" { | |
| endpoint = "localhost:4317" | |
| } | |
| exporter, err := otlptracegrpc.New(context.Background(), | |
| otlptracegrpc.WithEndpoint(endpoint), | |
| otlptracegrpc.WithInsecure(), | |
| ) | |
| exporter, err := otlptracegrpc.New(context.Background(), | |
| otlptracegrpc.WithInsecure(), | |
| ) |
| } | ||
| defer func() { _ = tp.Shutdown(context.Background()) }() | ||
|
|
||
| tracer := otel.Tracer("trace-generator") |
Member
There was a problem hiding this comment.
You could just use tp to avoid relying on global state
Suggested change
| tracer := otel.Tracer("trace-generator") | |
| tracer := tp.Tracer("trace-generator") |
| sdktrace.WithBatcher(exporter), | ||
| sdktrace.WithResource(res), | ||
| ) | ||
| otel.SetTracerProvider(tp) |
Member
There was a problem hiding this comment.
Not needed if you apply my last suggestion I think
Suggested change
| otel.SetTracerProvider(tp) |
Member
There was a problem hiding this comment.
I am pretty sure we could use telemetrygen for this but I'll leave that choice up to you
Comment on lines
+83
to
+89
| processors: [k8sattributes, batch] | ||
| exporters: [otlp] | ||
| # Host metrics with host-level resource attributes | ||
| metrics: | ||
| receivers: [hostmetrics] | ||
| processors: [batch] | ||
| exporters: [otlp] |
Member
There was a problem hiding this comment.
Continuation of previous-to-last comment
Suggested change
| processors: [k8sattributes, batch] | |
| exporters: [otlp] | |
| # Host metrics with host-level resource attributes | |
| metrics: | |
| receivers: [hostmetrics] | |
| processors: [batch] | |
| exporters: [otlp] | |
| processors: [k8sattributes] | |
| exporters: [otlp] | |
| # Host metrics with host-level resource attributes | |
| metrics: | |
| receivers: [hostmetrics] | |
| exporters: [otlp] |
| - job_name: 'kube-state-metrics' | ||
| scrape_interval: 15s | ||
| static_configs: | ||
| - targets: ['kube-state-metrics.otel-multi-pipeline-demo.svc.cluster.local:8080'] |
Member
There was a problem hiding this comment.
No strong opinion on this but we could use https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver instead
Comment on lines
+36
to
+38
| processors: | ||
| batch: | ||
| timeout: 10s |
Member
There was a problem hiding this comment.
Suggested change
| processors: | |
| batch: | |
| timeout: 10s |
| # App traces forwarded from DaemonSet agents (carry k8s.* attributes) | ||
| traces: | ||
| receivers: [otlp] | ||
| processors: [batch] |
Member
There was a problem hiding this comment.
Suggested change
| processors: [batch] |
| # AND kube-state-metrics scraped by the Prometheus receiver (carry k8s object-state labels) | ||
| metrics: | ||
| receivers: [otlp, prometheus] | ||
| processors: [batch] |
Member
There was a problem hiding this comment.
Suggested change
| processors: [batch] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Demonstrates how a standard Kubernetes OTel deployment (DaemonSet agent + Deployment gateway) produces three independent OTLP payload shapes — each carrying a different slice of host/workload identity:
Includes a Go trace-generator app, agent/gateway collector configs, kube-state-metrics deployment, Makefile for kind cluster setup, and a detailed README explaining exactly how and why the separation occurs.
What does this PR do?
Motivation