User Tracing

[mar-cf]

Overview

This stack adds a user-facing span pipeline to foundations, parallel to the existing internal tracing pipeline. Application code can emit spans into a separate USER_HARNESS that exports OTLP HTTP over a Unix domain socket (gRPC unsupported) to an OTLP endpoint, with per-trace routing metadata (an application-defined RoutingMetadata trait) carried on the wire and W3C traceparent continuation in/out.

The design is deliberately a mirror of internal tracing onto a second harness: every user API (user_span, start_user_trace, add_user_span_tags!, TelemetryContext.user_span, …) is the get_user() twin of an existing get() API. Two hard rules shape the surface:

• The user and internal pipelines are independent — separate harnesses, scope stacks, and exporters
• Public surface speaks W3C.

There are two layers to keep separate:

• init-time: stand up USER_HARNESS + the exporter, pointed at the OTLP endpoint's socket.
• per-request activation: start_user_trace(...) opens a root. Without a root, every user_span / with_user_span() / #[span_fn(user = true)] / add_user_span_tags! is a no-op.

---

User guide

Settings (init-time)

User tracing is configured via one optional block, gated by the user-tracing feature and fed to the existing foundations::telemetry::init:

TelemetrySettings.user_tracing: Option<UserTracingSettings>   // None = off

UserTracingSettings {
    enabled: bool,                       // default: true   (mirrors internal TracingSettings)
    max_queue_size: Option<NonZeroUsize>,// default: 1_000_000; None = unbounded (mirrors internal)
    output: UserTracesOutput,            // only variant today: OtlpUds(..)
}

OtlpUdsOutputSettings {
    socket_path: String,        // ← path to the OTLP endpoint's UDS (required, no default)
                                //   e.g. "/path/to/otlp-receptor.sock"
    routing_header_name: String,// ← header the endpoint reads each span's encoded routing from
                                //   (required, no default) e.g. "cf-trace-config"
    num_tasks: usize,           // default: 2    (concurrent export workers)
    max_batch_size: usize,      // default: 512  (spans drained per export batch)
}

The two settings a consumer must choose are socket_path (the OTLP endpoint's UDS) and routing_header_name (the header the endpoint reads routing from). enabled and max_queue_size mirror their internal-tracing equivalents; num_tasks and max_batch_size tune the exporter. All others have defaults and can be configured as needed.

There is deliberately no sampling configuration here. Unlike internal tracing, the user pipeline is not sampled inside foundations — the inbound user_tracing control header drives the activation (and therefore sampling) decision upstream.

Instrumentation APIs

Toy example covering the whole surface:

use foundations::telemetry::tracing::{
    self, RoutingMetadata, TraceparentContext, add_user_span_tags, user_span,
};

// `RoutingMetadata` is a trait — you define the concrete routing type. `group_key` batches
// spans that share it into one request; `encode` produces the routing header's value.
#[derive(Debug)]
struct MyRouting { zone_id: u64, account_id: u64 }

impl RoutingMetadata for MyRouting {
    fn group_key(&self) -> String { format!("{}|{}", self.zone_id, self.account_id) }
    fn encode(&self) -> String { format!("zone={};account={}", self.zone_id, self.account_id) }
}

// Per request: open the root. `routing` is required and fixed at construction (inherited by all
// descendants); `inbound` continues an upstream W3C trace, or None for a fresh one.
let _root = tracing::start_user_trace(
    "example_span_name",
    MyRouting { zone_id, account_id },
    inbound,                       // Option<TraceparentContext>
);

// Children — pick whichever fits:
let _child = user_span("lookup");                 // explicit standalone child
let _s     = tracing::span("db").with_user_span();// parallel user child *off an internal span*

#[tracing::span_fn("handle_request", user = true)]// whole-fn child (sync or async)
async fn handle_request() { /* ... */ }

// Annotate the current user span (no-op if no user trace is active):
add_user_span_tags!("cache.status" => "HIT");

// Across .await / spawn / separate hooks (UserSpanScope is !Send): hold the context.
let ctx = tracing::start_user_trace("req", routing, None).into_context(); // -> TelemetryContext
ctx.apply(async { let _c = user_span("work"); }).await;

// Outbound propagation to the next hop:
let traceparent: Option<String> = tracing::user_tracing::w3c_traceparent();

// Inbound parsing (strict W3C):
let inbound = TraceparentContext::parse(header_bytes); // -> Option<TraceparentContext>

Key points for users:

• The context carries the user span: into_context() / TelemetryContext::current() / #[span_fn] all propagate it across .await, even through an internal span's context — no manual threading.
• with_user_span() and #[span_fn(user = true)] open a user span in parallel with an internal span — this matches the guideline to create an internal span for every user span (the #[span_fn] decorator does the same), so a single call feeds both pipelines.

---

Notes to reviewers

The stack is four stacked PRs / seven commits. PR2–4 introduce the core user-span APIs; the surrounding PRs prepare for and power them — PR1 makes them safe to build, PR5–6 export them off-box, PR7 propagates them over W3C.

PR1 — user-tracing-1: Make SharedSpan construction explicit

Internal-only pre-move: replaces the blanket impl From<Span> (which always produced a tracked span) with an explicit shared_span() constructor, so user spans can later be built untracked and never enter the internal live registry.

PR2–4 — user-tracing-2-4: the in-process user API

Introduces the core and the ways to drive it:

• core: user_shared_span (untracked), start_user_trace / user_span, the UserSpanScope guard, the add_user_span_tags! / add_user_span_log_fields! / set_user_span_finish_callback! macros, get_user() + a dedicated USER_NOOP_HARNESS fallback, and the user_traces() test sink that observes user spans independently of the internal pipeline.
• propagation: TelemetryContext.user_span + UserSpanScope::into_context(), so a user span survives .await / spawn and rides along on an internal span's context.
• ergonomics: SpanScope::with_user_span() and #[span_fn(user = true)].

PR5–6 — user-tracing-5-6: ship spans off-box

• a temporary cf-rustracing [patch.crates-io] for the construction-time RoutingMetadata span property — a trait (group_key / encode) carried as Option<Arc<dyn RoutingMetadata>> (placeholder for a later version bump).
• the per-process pipeline (UserTracingSettings, init_user / USER_HARNESS, the OTLP/UDS exporter that groups spans by group_key and writes each group's encode() into the configured routing_header_name header) wired into telemetry::init, plus start_user_trace's required routing arg. Verified by producer tests that decode the exported OTLP body.

PR7 — user-tracing-7: W3C trace propagation

The TraceparentContext W3C parser, the optional inbound stitch on start_user_trace (continues the upstream trace — shared trace id, inbound parent), and user_tracing::w3c_traceparent() for outbound.

---

Alternatives considered

If constrained not to include user tracing specifics in this codebase, we can open up some seams to plug in similar functionality.

Routing metadata representation. cf-rustracing carries routing as Option<Arc<dyn RoutingMetadata>> — a small, Cloudflare-agnostic trait (group_key / encode). The concrete routing type (zone id / account id / account tag / destinations / persist) is defined by the consumer and never seen by cf-rustracing. Two alternatives were rejected:

• Concrete type in both repos. Define one concrete RoutingMetadata struct and share it as-is across cf-rustracing and foundations. Simplest, but it bakes Cloudflare routing concepts directly into cf-rustracing.
• Fully opaque Any. Carry routing as a type-erased Arc<dyn Any + Send + Sync> and downcast in the exporter. This also keeps Cloudflare types out of cf-rustracing, but trades the typed group_key / encode contract for runtime downcasting.

The trait keeps Cloudflare specifics out of cf-rustracing (like the opaque option) while giving the exporter a typed, downcast-free interface.

Pluggable exporter seam (BatchHandler). Rather than a concrete exporter inside foundations, foundations could accept an object-safe Arc<dyn BatchHandler> and run a shared drain loop, with the OTLP/UDS handler implemented in oxy:

// foundations: generic over the handler, free of OTLP/UDS deps
trait BatchHandler { /* process_batch(spans), initializer() */ }
init_user_harness(.., handler: Arc<dyn BatchHandler>);
// oxy: UserTracingUdsHandler implements BatchHandler

This would keep the OTLP/UDS (hyper / prost) deps and wire format out of foundations. We chose a concrete output module (output_otlp_uds, mirroring output_jaeger_thrift_udp / output_otlp_grpc): no trait object or plugin point — future destinations are new UserTracesOutput enum variants.

Additional notes

• Public surface speaks W3C; the jaeger SpanContext/SpanContextState conversion is internal-only — TraceparentContext is the only stitch type users see.
• The cf-rustracing [patch.crates-io] in PR5 is intentionally temporary; the matching change is a separate cf-rustracing PR and will become a normal version bump once released.

[mar-cf]

cloudflare/rustracing#11

[mar-cf]

@inikulin PTAL

[TheJokr]

please add references to the non-user macros that have more exhaustive docs. (you don't have to duplicate them.)

[TheJokr]

can this be an Option<UserSpanScope>?

Also: As far as I understand it, this is only used to enable a fluent API like span(...).with_user_tracing().into_context().apply(...). can we use a separate type for that instead so we dont pay the storage overhead on every SpanScope?

[TheJokr]

please mention here that this only forks the internal (non-user) trace

[TheJokr]

I think given how user-tracing is structured, there could be a user trace even if the regular trace was dropped due to sampling. In that case .operation_name() returns an empty string. How should we handle this?

[TheJokr]

please dont enable tracing by default. users have to configure a path for the UDS anyway, so they should also explicitly set enable: true.

we plan to turn off regular tracing by default as well: #158 (comment)

[TheJokr]

the default for our other exporters is 1, can you add a comment explaining why you chose a higher number for user tracing?

[TheJokr]

This channel uses a hardcoded gauge to record its current size. This needs to be changed to allow a gauge to be passed in. It's straightforward, and I don't this PR to become even larger, so I'll put up a quick PR tomorow.

[TheJokr]

nit: please use spans: &mut Vec<...> here and spans.drain(..) in the loop to avoid reallocating the vec in do_export every time

[TheJokr]

Would you be ok with moving the rest of the public freestanding functions in here as well? i mean:

• start_user_trace -> user_tracing::start_trace
• user_span -> user_tracing::span
• and maybe the macros as well

[TheJokr]

can you explain why you decided not to use https://docs.rs/cf-rustracing-jaeger/latest/cf_rustracing_jaeger/span/struct.SpanContextState.html?

it has a FromStr implementation to parse the context from a string, and we have the code in w3c_traceparent() to convert back into a string. what is missing, and could we more easily add it in cf-rustracing-jaeger?