fix: support standard OpenTelemetry tracing env vars

[AshuOPragmatikosThrylos]

Summary

This change resolves a configuration mismatch where the ratelimit service documented OpenTelemetry SDK environment variables but only honored custom TRACING_* variables for several critical tracing settings.

The service now:

• Accepts standard OTEL_* variables when the matching TRACING_* variable is not explicitly set
• Preserves full backward compatibility when TRACING_* is set
• Documents both configuration surfaces clearly in the README

Fixes #1166

---

Problem statement

Operators configuring ratelimit using the OpenTelemetry SDK environment variable spec expect settings like OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_SERVICE_NAME to work out of the box.

Before this change, only a subset of OTLP exporter settings worked (endpoint, TLS, timeout) because the underlying OTLP client reads those directly. Settings that pass through ratelimit's own settings.Settings layer were ignored unless duplicated into custom TRACING_* variables.

This created silent misconfiguration: tracing appeared configured, but protocol, enablement, service identity, or sampling did not behave as expected.

---

Root cause

Configuration was split across two layers with inconsistent env var support:

flowchart TB
    subgraph Before["Before fix"]
        A[Operator sets OTEL_* env vars] --> B[envconfig reads TRACING_* only]
        B --> C[ApplyOtelTracingFallbacks missing]
        C --> D[Runner uses partial TRACING_* values]
        D --> E[OTLP client reads endpoint/TLS from OTEL_*]
        D --> F[Protocol/name/sampling from wrong or default TRACING_* values]
        F --> G[Tracing misconfigured or disabled]
    end

Loading

Layer	What it read	Gap
settings.Settings via envconfig	TRACING_* only	Ignored OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_SERVICE_NAME, OTEL_TRACES_EXPORTER, sampler vars
trace.createClient() OTLP clients	Standard OTLP exporter env vars	Worked for endpoint/TLS/timeout only
README	Linked to OTEL spec + listed TRACING_*	Documented two incompatible mental models

---

Solution

Introduce ApplyOtelTracingFallbacks() immediately after envconfig.Process() in NewSettings():

flowchart TB
    subgraph After["After fix"]
        A[Process all env vars] --> B[envconfig loads TRACING_* defaults/explicit values]
        B --> C[ApplyOtelTracingFallbacks]
        C --> D{TRACING_* explicitly set?}
        D -->|Yes| E[Keep TRACING_* value]
        D -->|No| F[Resolve from matching OTEL_* var]
        E --> G[Runner initializes TracerProvider]
        F --> G
        G --> H[OTLP client exports with correct protocol + destination]
    end

Loading

Precedence rule

Priority	Source	Rationale
1 (highest)	Explicit TRACING_*	Backward compatibility for existing deployments
2	Matching OTEL_* fallback	Standards-compliant configuration for new deployments
3 (lowest)	Built-in defaults	Unchanged service defaults

---

Configuration mapping

Behavior	TRACING_* variable	OTEL_* fallback	Normalization
Enable tracing	TRACING_ENABLED	OTEL_TRACES_EXPORTER (otlp = on, none = off)	Respects OTEL_SDK_DISABLED=true
Exporter protocol	TRACING_EXPORTER_PROTOCOL	OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf → http, grpc → grpc
Service name	TRACING_SERVICE_NAME	OTEL_SERVICE_NAME	Trimmed string
Service namespace	TRACING_SERVICE_NAMESPACE	OTEL_RESOURCE_ATTRIBUTES	service.namespace=<value>
Service instance ID	TRACING_SERVICE_INSTANCE_ID	OTEL_RESOURCE_ATTRIBUTES	service.instance.id=<value>
Sampling rate	TRACING_SAMPLING_RATE	OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG	always_on→1.0, always_off→0.0, traceidratio→arg

Exporter destination settings (OTEL_EXPORTER_OTLP_ENDPOINT, certificates, timeout, headers) continue to be read directly by the OpenTelemetry OTLP client, unchanged.

---

Sequence: settings resolution at startup

sequenceDiagram
    participant Op as Operator
    participant Env as Environment
    participant NS as NewSettings()
    participant EC as envconfig.Process
    participant FB as ApplyOtelTracingFallbacks
    participant R as Runner

    Op->>Env: Set OTEL_EXPORTER_OTLP_PROTOCOL=grpc
    Op->>Env: Set OTEL_TRACES_EXPORTER=otlp
    Op->>Env: Set OTEL_SERVICE_NAME=my-ratelimit

    R->>NS: Load configuration
    NS->>EC: Parse TRACING_* struct tags
    EC-->>NS: Defaults (enabled=false, protocol=http, name=RateLimit)
    NS->>FB: Fill gaps from OTEL_* where TRACING_* unset
    FB->>Env: LookupEnv TRACING_* (not set)
    FB->>Env: Read OTEL_* values
    FB-->>NS: enabled=true, protocol=grpc, name=my-ratelimit
    NS-->>R: Final Settings
    R->>R: InitProductionTraceProvider(...)

Loading

---

Before vs after

Scenario	Before	After
OTEL_EXPORTER_OTLP_PROTOCOL=grpc only	Protocol stayed http (default)	Protocol resolves to grpc
OTEL_TRACES_EXPORTER=otlp only	Tracing stayed disabled	Tracing enabled
OTEL_SERVICE_NAME=my-ratelimit only	Service name stayed RateLimit	Service name resolves to my-ratelimit
OTEL_RESOURCE_ATTRIBUTES=service.namespace=prod,service.instance.id=pod-1	Namespace/instance empty	Both populated
OTEL_TRACES_SAMPLER=traceidratio, OTEL_TRACES_SAMPLER_ARG=0.25	Sampling stayed 1.0	Sampling resolves to 0.25
TRACING_EXPORTER_PROTOCOL=http + OTEL_EXPORTER_OTLP_PROTOCOL=grpc	N/A (OTEL ignored)	http wins (explicit TRACING_* precedence)

---

Files changed

File	Change
src/settings/tracing_config.go	New OTEL fallback resolver
src/settings/tracing_config_test.go	Unit tests for fallback + precedence
src/settings/settings.go	Invoke fallback after envconfig; clarify comments
README.md	Document both TRACING_* and OTEL_* variable sets

---

Test plan

Automated

go test ./src/settings/... -run "TestOtelTracing|TestTracingEnv" -count=1 -v
go test ./src/settings/... -count=1

Results (verified locally)

=== RUN   TestOtelTracingFallbacks_Protocol
--- PASS
=== RUN   TestOtelTracingFallbacks_ProtocolHttpProtobuf
--- PASS
=== RUN   TestOtelTracingFallbacks_ServiceName
--- PASS
=== RUN   TestOtelTracingFallbacks_ResourceAttributes
--- PASS
=== RUN   TestOtelTracingFallbacks_EnabledFromTracesExporter
--- PASS
=== RUN   TestOtelTracingFallbacks_DisabledFromSdkDisabled
--- PASS
=== RUN   TestOtelTracingFallbacks_SamplingRate
--- PASS
=== RUN   TestTracingEnvOverridesOtelEnv
--- PASS

ok   github.com/envoyproxy/ratelimit/src/settings   3.357s

Full settings package suite: 22/22 tests pass.

Runtime evidence (debug session, pre-instrumentation removal)

Observed in debug-b3e99e.log during verification:

Test case	Before	After	Evidence
OTEL_EXPORTER_OTLP_PROTOCOL=grpc	protocol=http	protocol=grpc	Log line 1
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf	protocol=http	protocol=http	Log line 2 (normalization)
OTEL_SERVICE_NAME=my-ratelimit	name=RateLimit	name=my-ratelimit	Log line 3
OTEL_RESOURCE_ATTRIBUTES=...	empty namespace/instance	prod / pod-123	Log line 4
OTEL_TRACES_EXPORTER=otlp	enabled=false	enabled=true	Log line 5
OTEL_SDK_DISABLED=true + exporter otlp	would enable	enabled=false	Log line 6
OTEL_TRACES_SAMPLER=traceidratio, arg 0.25	rate=1	rate=0.25	Log line 7
Explicit TRACING_* + conflicting OTEL	N/A	TRACING wins	Log line 8

---

Example configurations

Standards-first (recommended for new deployments)

export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
export OTEL_SERVICE_NAME=ratelimit
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=production,service.instance.id=${HOSTNAME}
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

Legacy-compatible (unchanged)

export TRACING_ENABLED=true
export TRACING_EXPORTER_PROTOCOL=grpc
export TRACING_SERVICE_NAME=ratelimit
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

---

Compatibility and risk

Concern	Assessment
Breaking existing deployments	No - explicit TRACING_* values still win
Silent behavior change	Low - only affects envs that set OTEL_* without matching TRACING_*
Scope of change	Settings layer only; no tracing span logic changes
Rollback	Revert commit; no migration required

---

FAQ (preemptive reviewer questions)

Q: Why not remove TRACING_* entirely?
A: Backward compatibility. Existing Helm charts, sidecars, and runbooks already use TRACING_*. Removing them would be a breaking change without a deprecation cycle.

Q: Why check os.LookupEnv("TRACING_*") instead of comparing against defaults?
A: Defaults like TRACING_EXPORTER_PROTOCOL=http are indistinguishable from an operator intentionally choosing HTTP. Explicit presence detection is the only safe precedence model.

Q: Why not use the OpenTelemetry SDK's auto-config env parser for everything?
A: Ratelimit still owns a service-specific enable gate and maps OTEL protocol values (http/protobuf) to its internal http/grpc switch. A thin fallback layer keeps behavior explicit and testable.

Q: Does this fix exporter endpoint configuration?
A: Endpoint configuration already worked via OTLP client env vars. This PR fixes the settings that were not delegated to the SDK.

Q: What about OTEL_TRACES_SAMPLER=parentbased_always_on and other samplers?
A: Not mapped in v1. Unsupported values are ignored; existing default sampling remains. Can be extended in a follow-up if needed.

Q: Does this address #326?
A: Partially related. #326 tracked adding tracing support (landed in #332). #1166 tracks configuration consistency with OTEL standards.

Q: Why fix README typo OLTP → OTLP?
A: Minor but reduces confusion while documenting the tracing section.

---

Suggested reviewer focus

1. Precedence logic in ApplyOtelTracingFallbacks() - confirm LookupEnv behavior matches intent
2. Protocol normalization table - confirm accepted OTEL values
3. README accuracy - confirm dual-config docs match implementation
4. Test coverage for override case (TestTracingEnvOverridesOtelEnv)

---

Checklist

• Root cause identified and documented
• Backward compatibility preserved
• Unit tests added
• Full settings package tests pass
• README updated
• Issue Inconsistent Tracing Configuration: Custom environment variables conflict with OpenTelemetry standards #1166 linked
• No debug instrumentation left in code

---

Related links

• Issue: Inconsistent Tracing Configuration: Custom environment variables conflict with OpenTelemetry standards #1166
• Related: Support Tracing #326
• OTEL SDK env spec: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/