Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ to include examples, links to docs, or any other relevant information.
This lets types with transfer type converters delegate their wire representation to the
configured payload converter, preserving SDK behavior such as serialization
contexts.
- Added `temporalio.contrib.opentelemetry.MetricsExporter`, which drains a
`temporalio.runtime.MetricBuffer` on a fixed interval and exports through a
real OpenTelemetry `MeterProvider`, giving SDK/Core metrics access to
standard OTel features (views, resource, exemplars). Experimental.
- Added `TLSConfig.verification_server_name` to verify the server certificate against a fixed name
instead of the connection's server name. Unlike `domain`, it does not change the TLS SNI or
HTTP/2 authority values, which keep following the connected host, so it can be used when the
Expand Down
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ informal introduction to the features and their implementation.
- [Observability](#observability)
- [Metrics](#metrics)
- [OpenTelemetry Tracing](#opentelemetry-tracing)
- [OpenTelemetry Metrics](#opentelemetry-metrics)
- [Protobuf 3.x vs 4.x](#protobuf-3x-vs-4x)
- [Known Compatibility Issues](#known-compatibility-issues)
- [gevent Patching](#gevent-patching)
Expand Down Expand Up @@ -1990,6 +1991,40 @@ as an interceptor on the `interceptors` argument of `Client.connect`. When set,
calls and for all activity and workflow invocations on the worker, spans will be created and properly serialized through
the server to give one proper trace for a workflow execution.

#### OpenTelemetry Metrics

Metrics support also requires the `opentelemetry` extra (see above). If you want your Temporal SDK/Core metrics (and
any custom metrics recorded via `activity.metric_meter()`/`workflow.metric_meter()`) to flow through the standard
OpenTelemetry metrics pipeline (views, resources, any OTel-compatible backend) rather than only through
`PrometheusConfig`/`OpenTelemetryConfig`, set a `temporalio.runtime.MetricBuffer` as the `metrics` on
`TelemetryConfig`, then drain it into a real OpenTelemetry `MeterProvider` using
`temporalio.contrib.opentelemetry.MetricsExporter`:

```python
from datetime import timedelta

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader

from temporalio.client import Client
from temporalio.contrib.opentelemetry import MetricsExporter
from temporalio.runtime import MetricBuffer, Runtime, TelemetryConfig

buffer = MetricBuffer(10_000)
runtime = Runtime(telemetry=TelemetryConfig(metrics=buffer))
meter_provider = MeterProvider(
metric_readers=[PeriodicExportingMetricReader(ConsoleMetricExporter(), export_interval_millis=5000)]
)

async with MetricsExporter(buffer, meter_provider):
client = await Client.connect("localhost:7233", runtime=runtime)
# ... run workers/workflows while the exporter drains the buffer in the background
```

`MetricsExporter` must be running (via `async with` or manual `run()`/`shutdown()`) for as long as metrics should be
exported, since it works by draining the buffer on a fixed interval (`poll_interval`, default one second) -- per the
warning on `MetricBuffer`, updates are dropped if the buffer isn't drained regularly.

### Protobuf 3.x vs 4.x

Python currently has two somewhat-incompatible protobuf library versions - the 3.x series and the 4.x series. Python
Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ classifiers = [

[project.optional-dependencies]
grpc = ["grpcio>=1.48.2,<2"]
opentelemetry = ["opentelemetry-api>=1.11.1,<2", "opentelemetry-sdk>=1.11.1,<2"]
opentelemetry = ["opentelemetry-api>=1.12.0,<2", "opentelemetry-sdk>=1.12.0,<2"]
Comment thread
harsh543 marked this conversation as resolved.
pydantic = ["pydantic>=2.0.0,<3"]
openai-agents = ["openai-agents>=0.17.5", "mcp>=1.9.4, <2"]
google-adk = ["google-adk>=2.2.0,<3", "mcp>=1.24,<2"]
Expand All @@ -38,8 +38,8 @@ deepagents = [
"langchain-core>=1.4.8,<2; python_version >= '3.11'",
]
lambda-worker-otel = [
"opentelemetry-api>=1.11.1,<2",
"opentelemetry-sdk>=1.11.1,<2",
"opentelemetry-api>=1.12.0,<2",
"opentelemetry-sdk>=1.12.0,<2",
"opentelemetry-exporter-otlp-proto-grpc>=1.11.1,<2",
"opentelemetry-semantic-conventions>=0.40b0,<1",
"opentelemetry-sdk-extension-aws>=2.0.0,<3",
Expand Down
25 changes: 24 additions & 1 deletion temporalio/contrib/opentelemetry/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,29 @@
# OpenTelemetry Integration for Temporal Python SDK

This package provides OpenTelemetry tracing integration for Temporal workflows, activities, and other operations. It includes automatic span creation and propagation for distributed tracing across your Temporal applications.
This package provides OpenTelemetry tracing and metrics integration for Temporal workflows, activities, and other operations. It includes automatic span creation and propagation for distributed tracing, and a `MetricsExporter` for exporting Temporal SDK/Core metrics through the standard OpenTelemetry metrics API, across your Temporal applications.

## Metrics

`MetricsExporter` drains a `temporalio.runtime.MetricBuffer` into an OpenTelemetry `MeterProvider`, so Temporal's own SDK/Core metrics (and any custom metrics recorded via `activity.metric_meter()`/`workflow.metric_meter()`) can be exported through the standard OpenTelemetry metrics pipeline (views, resources, any OTel-compatible backend) instead of only through `PrometheusConfig`/`OpenTelemetryConfig`.

```python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from temporalio.contrib.opentelemetry import MetricsExporter
from temporalio.runtime import MetricBuffer, Runtime, TelemetryConfig

buffer = MetricBuffer(10_000)
runtime = Runtime(telemetry=TelemetryConfig(metrics=buffer))
meter_provider = MeterProvider(
metric_readers=[PeriodicExportingMetricReader(ConsoleMetricExporter())]
)

async with MetricsExporter(buffer, meter_provider):
client = await Client.connect("localhost:7233", runtime=runtime)
...
```

**Note:** the `Runtime` must be constructed with the buffer attached *before* `MetricsExporter` is started, and the exporter must keep running (it polls on a fixed interval) for as long as metrics should be exported.

## Overview

Expand Down
10 changes: 7 additions & 3 deletions temporalio/contrib/opentelemetry/__init__.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
"""OpenTelemetry v2 integration for Temporal SDK.

This package provides OpenTelemetry tracing integration for Temporal workflows,
activities, and other operations. It includes automatic span creation and
propagation for distributed tracing.
This package provides OpenTelemetry tracing and metrics integration for
Temporal workflows, activities, and other operations. It includes automatic
span creation and propagation for distributed tracing, and a
:py:class:`MetricsExporter` for exporting Temporal SDK/Core metrics through
the standard OpenTelemetry metrics API.
"""

from temporalio.contrib.opentelemetry._interceptor import (
TracingInterceptor,
TracingWorkflowInboundInterceptor,
)
from temporalio.contrib.opentelemetry._metrics_exporter import MetricsExporter
from temporalio.contrib.opentelemetry._otel_interceptor import OpenTelemetryInterceptor
from temporalio.contrib.opentelemetry._plugin import OpenTelemetryPlugin
from temporalio.contrib.opentelemetry._tracer_provider import create_tracer_provider
Expand All @@ -18,5 +21,6 @@
"TracingWorkflowInboundInterceptor",
"OpenTelemetryInterceptor",
"OpenTelemetryPlugin",
"MetricsExporter",
Comment thread
harsh543 marked this conversation as resolved.
"create_tracer_provider",
]
Loading
Loading