feat: promote allowlisted caller context onto every agent span

[rtemperini]

Motivation

Agent spans record what an agent did but not who asked for it. Once a request
enters kagent the caller's identity and context are gone, so traces cannot be
filtered or grouped by the calling user, the conversation thread, or the ticket
that triggered the run.

Two common deployment shapes hit this:

• An authenticating proxy in front of kagent knows the signed-in user's email
  from an OIDC token. That email should appear on every downstream operation —
  tool calls, A2A delegations, MCP calls, model calls.
• A chat integration or other programmatic A2A caller knows the invoking user,
  the thread, and the channel. None of it reaches the trace.

#1734 raised the second case
and was closed by #1737, which
promotes A2A message.metadata into a2a.message.metadata.* attributes. That
left two gaps:

1. Go only. The Python runtimes ignore inbound message.metadata entirely.
2. Root span only. SetMessageMetadataAttributes writes to the span current
   at A2A entry, so descendant spans inherit nothing.

Gap 2 is the one that breaks the use case: Langfuse and comparable backends
resolve trace-level filters against the attributes present on each span, so an
attribute on the invocation span alone leaves most views unfilterable.

What this changes

An operator names the context keys they want traced. Both runtimes read those
keys from W3C Baggage and from A2A message.metadata, sanitise them, and
merge them into the request-scoped attribute bag that
KagentAttributesSpanProcessor / kagentAttributesSpanProcessor already stamps
onto every span of the request.

otel:
  tracing:
    enabled: true
    contextKeys: [user.email, user.name, thread_id, channel]

baggage: user.email=ada@example.com   →  kagent.context.user.email = "ada@example.com"
metadata: {"thread_id": "1717171.42"} →  kagent.context.thread_id  = "1717171.42"

Adding a new traced value is a configuration change, not a code change.

Design rationale

Why baggage

Baggage is the vendor-neutral OTel answer to this problem, and the plumbing
already exists: the controller, both runtimes, and every instrumented HTTP client
run a composite tracecontext + baggage propagator. A value set once at the edge
survives controller → agent → sub-agent → tool without kagent adding any
hop-specific mechanism, and it requires no kagent-specific knowledge from the
caller — any OTel SDK or proxy can set it.

A2A message.metadata remains supported as the complement, for callers that can
set message fields but not transport headers. Because it is scoped to one message
it is the more specific source, so it wins on conflict.

Alternatives considered and rejected: a fixed set of user/thread/channel
fields (not extensible, needs a code change per new field); custom HTTP headers
(reinvents baggage, does not cross hops); stamping only the root span (does not
satisfy the per-span requirement above).

Why the attributes go in the request-scoped bag

The bag is the only place where a value is applied by the span processor at
OnStart for every span, including spans created by upstream ADK, the MCP
client, and the model instrumentation — code kagent does not own and cannot
instrument individually.

Why one knob instead of an enable flag plus a list

An enabled: true with an empty allowlist is a state that does nothing but looks
like it should. Making the allowlist itself the switch removes that state: empty
means off, non-empty means on, and a contradiction cannot be expressed.

Feature flag

Name	KAGENT_TRACE_CONTEXT_KEYS (env) / otel.tracing.contextKeys (Helm)
Type	Comma-separated allowlist / list of strings
Default	Empty — promotion disabled

The controller forwards the variable to the agents it creates. It needs explicit
forwarding because collectOtelEnvFromProcess carries only OTEL_ prefixed
names.

Which caller data reaches a trace backend is cluster-wide operator policy, so the
value is applied after the Harness environment and any inherited entry of the
same name is dropped first. Without that second step a Harness could enable
promotion whenever the operator had configured nothing at all; there is a test
for exactly that.

Security considerations

Caller-supplied context is untrusted input on both paths, so promotion is
constrained on every axis:

Risk	Control
Cardinality explosion	Only allowlisted keys are read; the allowlist is capped at 32 entries
Oversized spans	Values truncated to 256 characters, keys to 64 (rune-based on both runtimes)
Log / trace injection	Control characters stripped from values
Shadowing semantic conventions	Every attribute is namespaced under kagent.context., so service.name and friends cannot be overwritten even if an operator allowlists them — structural, not a denylist
Secret leakage	Nothing is promoted unless an operator names the key; raw values are never logged
Malformed attribute names	Allowlist entries containing whitespace or control characters are dropped
A tenant widening the allowlist	Operator-level configuration only; a same-named Harness env entry is dropped, not inherited

Non-scalar metadata (objects, arrays) is skipped: unbounded in size, meaningless
as an attribute value.

The docs state plainly that anything allowlisted is visible to everyone with
access to the trace backend, and that callers control the values.

Backwards compatibility

Fully backwards compatible.

• Default is off. With no contextKeys set, the ConfigMap key is absent, the env
  var is unset, the helper returns immediately, and not a single span attribute
  changes.
• No existing behaviour is modified or removed. The a2a.message.metadata.*
  attributes from feat(go-adk): propagate A2A message metadata as OTEL span attributes #1737 are untouched and still unconditional in the Go runtime.
• No new dependencies. Baggage comes from go.opentelemetry.io/otel and
  opentelemetry-api, both already required.
• No API, CRD, or protobuf changes.

Runtime parity

The Go and Python implementations share the same allowlist parsing, the same
baggage-then-metadata precedence, the same limits, the same control-character
definition (Python's _is_control deliberately matches Go's
unicode.IsControl), rune-based rather than byte-based truncation on both sides,
and the same kagent.context. namespace. Both are covered by equivalent test
cases so the two cannot drift silently.

The ADK, LangGraph, and CrewAI Python executors all promote context. Reading a
Message's protobuf Struct metadata was extracted into
kagent.core.a2a.read_message_metadata rather than repeated three times.

Testing

Added, in both runtimes:

• flag off → no behaviour change (asserted against exported spans, not just the
  helper's return value)
• flag on, baggage path
• flag on, A2A message.metadata path
• metadata overrides baggage on conflict
• keys outside the allowlist ignored
• scalar rendering (string / bool / int / float, including the protobuf Struct
  float-integer case) and non-scalar skipping
• control-character stripping, value truncation, key-length and duplicate
  rejection, allowlist cap
• semantic conventions cannot be shadowed
• attribute-on-every-span: a root → tool → model span tree, asserting the
  attribute is present on all three

Plus: Helm unittest for ConfigMap rendering with and without contextKeys; Go
tests that the controller forwards the variable and that a Harness can neither
widen nor enable the allowlist; and a regression test that the OTel Python SDK's
default propagator still carries baggage, since the Python runtime relies on that
default rather than configuring a propagator.

Run locally against this branch:

Suite	Result
make -C go lint	0 issues
go test -race (5 touched packages)	136 passed, 0 failed
ruff format --diff	176 files already formatted
ruff check (files touched here)	all checks passed
pytest ./packages/*/tests	489 passed, 1 skipped, 1 pre-existing failure
helm unittest helm/kagent	21 suites, 276 passed

The one Python failure is test_tls_e2e.py::test_e2e_with_system_and_custom_ca,
which fails identically on an unmodified main checkout (1 failed, 10 passed, 1 skipped on both). A full go test -race -skip 'TestE2E.*' ./... also hits
TestFetchSourceReusesExistingMaterialization in core/v2/agentplugins, a macOS
/private/var vs /var symlink artefact that likewise fails unmodified on
main. Neither package is touched by this PR.

Not in scope

• E2E coverage. The behaviour is span-attribute shaping with no CRD, API, or
  lifecycle surface, and it is fully covered by unit tests against a real
  TracerProvider. Happy to add an E2E case if maintainers would prefer one.
• Configurable attribute namespace. The fixed kagent.context. prefix is
  what makes shadowing structurally impossible. Backends that need their own
  names should rename in the OTel Collector; the docs include a recipe.
• Controller-side baggage injection. The controller already propagates
  baggage it receives. Deriving baggage from an OIDC token at the edge is the
  gateway's job, not kagent's.

Docs

New docs/architecture/trace-context.md,
linked from the architecture index. Covers configuration, why baggage, the
per-span guarantee, the safety properties, and the Collector rename recipe.

Commits

feat(adk)	Go runtime: allowlist, sanitisation, A2A executor wiring
feat(python)	Python runtimes: parity across ADK, LangGraph, CrewAI
feat(core)	Helm value, ConfigMap, controller forwarding
docs(architecture)	Documentation

All four commits are DCO signed off.

This follows on from #1734 / #1737 rather than starting a new discussion, but
happy to write it up as an enhancement proposal under design/ first if that is
preferred for a change of this size.

[krisztianfekete]

Thanks, added a few initial comments!

[krisztianfekete]

Totally agree with the goal, but as per the OTel's recommendations: https://opentelemetry.io/docs/security/handling-sensitive-data/, email addresses and names should never be attributes at all.

OIDC already hands you an opaque sub, which is what user.id should use. Could we make sub the example instead of the email?

[krisztianfekete]

Let's use subs everywhere as exampels as mentioned in the md file above.

[krisztianfekete]

Same as above, let's not document it like this as it's an anti-pattern.

[krisztianfekete]

Again, emails and username should not be here as per the guidance because it's a compliance question. Operators should also not do this. Also baggage is on HTTP headers, and today this data goes to api.openai.com and every HTTP MCP server too.

[krisztianfekete]

If you don't want to lose the OIDC format the recommendation is to hash it, and user.hash is a registry attribute that exists for exactly that. Something like:

contextKeys:
  - {from: sub,       to: user.id}
  - {from: email,     to: user.hash, hash: hmac-sha256}
  - {from: thread_id, to: kagent.thread_id}

[krisztianfekete]

The prefix is right for custom keys e.g. channel, but it also blocks the names that do exist in the registry. user.id, enduser.id, session.id are all real attributes, and we should use the semconv names before inventing new ones.

Could we let a small fixed set through unprefixed (user.*, enduser.*, session.id) and prefix everything else?

[krisztianfekete]

Also reflect this in tests.