Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
b1ef021
docs: design Kairo guardrail span sample
nikhilc-microsoft Aug 3, 2026
7e671e2
feat: add Kairo sample input guardrail
nikhilc-microsoft Aug 3, 2026
71b043d
test: log Kairo guardrail export payload
nikhilc-microsoft Aug 3, 2026
b83760c
fix: address Task 2 review findings
nikhilc-microsoft Aug 3, 2026
a8e3509
feat: instrument Kairo sample guardrail span
nikhilc-microsoft Aug 3, 2026
4fc8d66
fix: address Task 3 review findings
nikhilc-microsoft Aug 3, 2026
f2e1b27
chore: remove temporary guardrail artifacts
nikhilc-microsoft Aug 3, 2026
8b05b70
Merge branch 'main' of https://github.com/microsoft/opentelemetry-dis…
nikhilc-microsoft Aug 12, 2026
b1c9c46
Merge branch 'main' of https://github.com/microsoft/opentelemetry-dis…
nikhilc-microsoft Aug 12, 2026
d6f678d
Document A365 durable delivery design
nikhilc-microsoft Aug 12, 2026
0931b1f
Plan A365 durable delivery implementation
nikhilc-microsoft Aug 12, 2026
4e16524
Ignore local worktrees
nikhilc-microsoft Aug 12, 2026
b8f06a7
Add A365 transmission gate
nikhilc-microsoft Aug 12, 2026
f493aa9
Add secure A365 persistent storage
nikhilc-microsoft Aug 12, 2026
f8de803
Fix Task 2 review findings in persistent storage
nikhilc-microsoft Aug 12, 2026
e5ddf85
Append Task 2 fix report to task-2-report.md
nikhilc-microsoft Aug 12, 2026
3a03a72
Remove SDD scratch report
nikhilc-microsoft Aug 12, 2026
36a9f50
Add A365 durable replay coordinator
nikhilc-microsoft Aug 12, 2026
4f16ef4
Fix Task 3 review: explicit ReplayIdentityError, general-exception st…
nikhilc-microsoft Aug 12, 2026
6c34586
Add durable delivery to A365 exporter
nikhilc-microsoft Aug 12, 2026
315aea6
Expose A365 offline storage options
nikhilc-microsoft Aug 12, 2026
dea826c
Harden A365 durable delivery from review notes
nikhilc-microsoft Aug 13, 2026
9691e11
Fix durable-queue capacity wedge and harden storage file permissions
nikhilc-microsoft Aug 13, 2026
1200b44
fix: clamp backoff exponent and harden replay thread loop
nikhilc-microsoft Aug 13, 2026
822b544
Document A365 durable delivery hardening
nikhilc-microsoft Aug 14, 2026
ffd3607
Plan A365 durable delivery hardening
nikhilc-microsoft Aug 14, 2026
0f9b04d
Harden A365 retry timing and transmission gate
nikhilc-microsoft Aug 14, 2026
8021aaa
Harden durable schema and replay replay-endpoint security
nikhilc-microsoft Aug 14, 2026
faf746b
Harden A365 replay terminal accounting
nikhilc-microsoft Aug 14, 2026
9e47d71
Add batch regression coverage for replay deletes
nikhilc-microsoft Aug 14, 2026
bcf4065
Stop tracking SDD task report
nikhilc-microsoft Aug 14, 2026
5b9a019
Remove durable delivery scratch report
nikhilc-microsoft Aug 14, 2026
2c3b624
Isolate A365 permanent failures and validate batching
nikhilc-microsoft Aug 14, 2026
6c00ffb
Add drain-safe A365 batch processing
nikhilc-microsoft Aug 14, 2026
43a14bb
Fix Task 5 review findings in _EnrichingBatchSpanProcessor
nikhilc-microsoft Aug 14, 2026
acae1de
Make A365 exporter shutdown deterministic
nikhilc-microsoft Aug 14, 2026
9d087ef
Fix Task 6 shutdown hardening follow-up
nikhilc-microsoft Aug 14, 2026
ab4174b
Validate A365 durable delivery hardening
nikhilc-microsoft Aug 14, 2026
89a050b
Fix final durable delivery review findings
nikhilc-microsoft Aug 14, 2026
1bc63f9
Remove unrelated Kairo sample artifacts
nikhilc-microsoft Aug 14, 2026
225a8e7
Keep durable delivery PR scope focused
nikhilc-microsoft Aug 14, 2026
0ef5e66
Fix A365 batch processor lint
nikhilc-microsoft Aug 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions A365_DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ use_microsoft_opentelemetry(
| `a365_scheduled_delay_ms` | `int` | `5000` | — | Delay between A365 export batches in milliseconds. |
| `a365_exporter_timeout_ms` | `int` | `30000` | — | Timeout for a single A365 export operation in milliseconds. |
| `a365_max_export_batch_size` | `int` | `512` | — | Maximum batch size for a single A365 export operation. |
| `a365_exporter_disable_offline_storage` | `bool` | `False` | — | Disable durable offline storage. When `True`, failed exports are not persisted to disk and at-least-once delivery is not guaranteed. Defaults to `False` (storage enabled). |
| `a365_exporter_storage_directory` | `str` | `None` | — | Custom directory for durable offline storage. When `None`, a platform default path is used. Choose a path that only the current user or service account can read, because stored OTLP payloads may contain prompts or completions when sensitive-data capture is enabled. |

### Resource / Service Name

Expand Down Expand Up @@ -83,6 +85,52 @@ use_microsoft_opentelemetry(
)
```

### Durable Delivery (Offline Storage)

When `a365_enable_observability_exporter=True`, the A365 exporter persists failed export payloads to disk and replays them once connectivity is restored — providing **at-least-once delivery** semantics.

Key defaults and limits:

| Property | Value |
|---|---|
| Enabled by default | Yes (when exporter is active) |
| Retention window | 2 days |
| Maximum storage size | 50 MB |
| Storage path | Platform default (see below) |

The default path is a sub-directory of the platform's local app data folder,
derived from the user, executable, and working directory. Processes for the
same application can share the SQLite queue; leases coordinate replay across
those processes.

**Replay behavior:**

- Each queued record stores only its identity (tenant, agent, agentic user, S2S flag) and payload — not a fixed URL. On replay, the exporter reconstructs the export endpoint from the exporter's *current* configuration and re-resolves a fresh bearer token, so records queued before an endpoint or credential change are still delivered correctly, and any endpoint that would not resolve to HTTPS is rejected rather than replayed.
- Records that are permanently rejected by the service (e.g. `400 Bad Request`) or that fail internal validation (unsupported schema version, blank tenant/agent id, or blank payload — "poison" records) are discarded from the durable queue instead of being retried forever.
- `shutdown()` is drain-safe: it signals the background replay loop to stop and blocks until any in-flight replay send actually finishes before closing the durable store and HTTP session, so an in-progress send is never left holding a closed resource and no accepted span is dropped mid-flight. Concurrent `shutdown()` callers all wait for that same cleanup to complete.

**To disable durable storage** (no disk writes, best-effort delivery only):

```python
use_microsoft_opentelemetry(
enable_a365=True,
a365_enable_observability_exporter=True,
a365_exporter_disable_offline_storage=True,
)
```

**To specify a custom storage directory:**

```python
use_microsoft_opentelemetry(
enable_a365=True,
a365_enable_observability_exporter=True,
a365_exporter_storage_directory="/var/lib/my-agent/telemetry",
)
```

> **Security note:** Stored OTLP payloads are unencrypted. When sensitive-data capture is enabled (`enable_sensitive_data=True`), payloads may include prompts, completions, or tool arguments. Restrict the storage directory to the service account running the agent (e.g., `chmod 700`).

## Auto-Instrumented Libraries

It auto-discovers and activates supported instrumentors via OpenTelemetry entry points. No manual `instrument()` calls are needed.
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@
### Features Added
- Add support for agent identity propagation for compiled agents in nested graph
([#245](https://github.com/microsoft/opentelemetry-distro-python/pull/245))
- Expose A365 offline storage options: `a365_exporter_disable_offline_storage` (default `False`)
and `a365_exporter_storage_directory` (default `None`) on `use_microsoft_opentelemetry`,
`Agent365ExporterOptions`, and `create_a365_components`.
When `a365_enable_observability_exporter=True`, the exporter provides at-least-once delivery
by persisting failed payloads to disk (up to 2 days / 50 MB) and replaying them on recovery.
Set `a365_exporter_disable_offline_storage=True` to opt out.
Stored payloads are unencrypted; restrict the storage path to the service account, especially
when `enable_sensitive_data=True` (payloads may contain prompts or completions).
Replay reconstructs the current HTTPS export endpoint and bearer token at send time rather
than reusing a stale URL, permanently-rejected or malformed ("poison") records are discarded
instead of retried forever, and `shutdown()` is drain-safe: it waits for any in-flight replay
send to finish before closing the durable store and HTTP session (parity with `.NET` PR #137).

# 1.3.7 (2026-08-05)
### Features Added
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ See the [A365 guide](https://github.com/microsoft/opentelemetry-distro-python/bl
| `a365_scheduled_delay_ms` | `int` | `5000` | Delay between A365 export batches (ms). |
| `a365_exporter_timeout_ms` | `int` | `30000` | Timeout for a single A365 export operation (ms). |
| `a365_max_export_batch_size` | `int` | `512` | Maximum batch size for a single A365 export operation. |
| `a365_exporter_disable_offline_storage` | `bool` | `False` | Disable durable offline storage. When `True`, failed exports are not persisted to disk. |
| `a365_exporter_storage_directory` | `str` | `None` | Custom directory for durable offline storage. Restrict to the service account; payloads may contain prompts or completions when sensitive-data capture is enabled. |

> For A365 token resolver patterns, baggage, and scope classes, see the [A365 guide](https://github.com/microsoft/opentelemetry-distro-python/blob/main/A365_DOCUMENTATION.md).

Expand Down
Loading
Loading