Skip to content

Export span links through native transport - #6129

Open
lloeki wants to merge 13 commits into
masterfrom
lloeki/native-span-links-consumer
Open

Export span links through native transport#6129
lloeki wants to merge 13 commits into
masterfrom
lloeki/native-span-links-consumer

Conversation

@lloeki

@lloeki lloeki commented Jul 30, 2026

Copy link
Copy Markdown
Member

AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

What does this PR do?

Exports canonical SpanLink#to_hash payloads through the native transport, including 128-bit IDs, flags, tracestate, flattened attributes, and link order.

The conversion snapshots links and their string data before native allocation so Ruby mutation and default-proc callbacks cannot invalidate borrowed memory.

Motivation:

The native exporter does not otherwise carry span links. This adds payload parity for distributed tracing and OpenTelemetry link data.

Tracked by APMSP-3219.

Change log entry

Yes. Native trace export now supports span links.

Additional Notes:

dropped_attributes_count is intentionally deferred by additive commit 1893676d. The Agent v0.4 and indexed V1 schemas do not define it, and the existing Ruby field does not produce a non-zero value. Reverting that commit restores the native wiring if a protocol field is introduced.

Current stack position: R5 / span links, based on the common v39 uptake in #6143. M1 and events are sibling consumers; M2 follows M1, and M3 follows M2.

Stage libdatadog API dd-trace-rb consumer
Runtime and panic containment #2302 #6143
M1 opaque meta_struct blobs #2303 #6130
Span links #2305 #6129
Span events #2301 #6134
M2 native structured values #2304 #6132
M3 direct structured setter #2300 #6133

Independent fork and exception-safety prerequisites are #6127 and #6128.

How to test the change?

  • Native transport suite: 145 examples, 0 failures against published v39, on Ruby 2.5 and 4.0.
  • Span-link and distributed propagation coverage: 85 examples, 0 failures, 1 existing pending example.
  • OpenTelemetry suite: 248 examples, 0 failures.
  • Standard and Steep passed.

@lloeki lloeki added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jul 30, 2026
@datadog-official

datadog-official Bot commented Jul 30, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 90.25% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: adab38d | Docs | View more details | Give us feedback!

@dd-octo-sts dd-octo-sts Bot added the tracing label Jul 30, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Thank you for updating Change log entry section 👏

Visited at: 2026-07-30 12:12:07 UTC

@lloeki
lloeki force-pushed the lloeki/native-libdatadog-next-uptake branch from 39c86b1 to 6fe683b Compare July 30, 2026 12:01
@lloeki
lloeki force-pushed the lloeki/native-span-links-consumer branch from 9e873ee to 20b9a60 Compare July 30, 2026 12:02
@lloeki
lloeki force-pushed the lloeki/native-span-links-consumer branch from 20b9a60 to 1d4a188 Compare August 3, 2026 14:54
@lloeki
lloeki changed the base branch from lloeki/native-libdatadog-next-uptake to lloeki/native-libdatadog-payload-api-uptake August 3, 2026 14:55
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Ensures the combined libdatadog profiling artifact enables `catch_panic` for data-pipeline FFI entry points. It also adds a packaged C regression that triggers capacity overflow and verifies that the FFI returns an error instead of aborting the process.

# Motivation

The standalone data-pipeline FFI crate enables panic containment by default, but the combined profiling artifact disables dependency defaults and did not re-enable it. This closes [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830) before the native trace exporter payload APIs ship together.

# Additional Notes

**Stack position: L0, current PR: [DataDog/libdatadog PR 2302](#2302

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| **L0 (current)** | **[DataDog/libdatadog PR 2302](#2302 | Release prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo test -p libdd-profiling-ffi --features data-pipeline-ffi` passed the combined profiling/data-pipeline FFI suite: 57 tests.
- `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic trace_chunks_new_with_overflow_capacity_returns_panic_error` passed.
- `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic begin_chunk_with_overflow_capacity_returns_panic_error` passed.
- `nix develop -c cargo ffi-test --filter trace_exporter_panic_containment` built and passed the packaged C panic-containment regression.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- `nix develop -c cargo clippy -p libdd-profiling-ffi --all-targets --features data-pipeline-ffi -- -D warnings` passed.




[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Co-authored-by: hoolioh <julio.gonzalez@datadoghq.com>
@lloeki
lloeki force-pushed the lloeki/native-libdatadog-payload-api-uptake branch from a7ef512 to 703d524 Compare August 4, 2026 11:19
lloeki added 4 commits August 4, 2026 13:20
Normalize each link through `SpanLink#to_hash`, validate its canonical
fields, and pass the complete ordered link batch to libdatadog. This
keeps OpenTelemetry and distributed propagation semantics consistent
when traces use the native transport.

Stop warning about span links now that their IDs, attributes,
tracestate, and flags are preserved on the wire.
Canonical link hashes may use stateful defaults or mutate borrowed Ruby
strings. Copy every field and string byte exactly once before allocating
the Rust span so later Ruby callbacks cannot invalidate FFI pointers or
produce an internally inconsistent link.

Own partial snapshots explicitly so exceptions during normalization
release every C allocation before propagating back to Ruby.
Preserve `dropped_attributes_count` from the canonical link format so
native exports retain the complete link contract.

Delay borrowing scalar span string pointers until link normalization
has finished because `SpanLink#to_hash` can run Ruby code that mutates
or relocates those strings.
Remove native extraction and FFI forwarding for
`dropped_attributes_count`. Neither the Agent v0.4 nor indexed V1
span-link schema defines this field, and current Datadog tracers do not
produce non-zero counts.

The historical pure-Ruby `SpanLink#to_hash` scaffold remains unchanged.
This additive commit can be reverted when the Agent protocols formally
allocate and support the field.
@lloeki
lloeki force-pushed the lloeki/native-span-links-consumer branch from 1893676 to 3bba941 Compare August 4, 2026 11:20
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Adds an opaque per-key `meta_struct` blob setter to `TracerSpan`. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model.

# Motivation

Native exporter consumers cannot currently populate `meta_struct`, which blocks payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). The opaque setter provides the safe first stage before the language-neutral structured-value APIs.

# Additional Notes

**Stack position: L1, current PR: [DataDog/libdatadog PR 2303](#2303

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer |
| **L1 (current)** | **[DataDog/libdatadog PR 2303](#2303 | **[R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130 |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys.
- `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed.
- `nix develop .#nightly -c cargo fmt --all -- --check` passed.
- The generated C header was rebuilt through `nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux` and consumed successfully by the matching dd-trace-rb branch.
- `nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb` passed 116 examples, including AppSec and AI Guard-shaped wire conformance.
- Targeted Standard and Steep checks for the native transport passed.




[APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
@lloeki
lloeki marked this pull request as ready for review August 4, 2026 14:08
@lloeki
lloeki requested review from a team as code owners August 4, 2026 14:08
@lloeki
lloeki requested review from mabdinur and marcotc and removed request for a team August 4, 2026 14:08
@p-datadog
p-datadog requested a review from Copilot August 4, 2026 19:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds native-transport support for exporting span links by snapshotting canonical SpanLink#to_hash data in the C extension and wiring it into ddog_tracer_span_set_links, with accompanying spec updates to validate wire conformance, ordering, and mutation-safety.

Changes:

  • Add span-link snapshotting and FFI conversion in ext/libdatadog_api/trace_exporter.c, then set links on the Rust span during native conversion.
  • Update native transport warning logic to stop warning on span links (now supported) and adjust related transport specs.
  • Add/extend conformance and integration specs to assert canonical link hashes, ordering, and behavior under mutation/default-proc hazards.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
spec/datadog/tracing/transport/native/transport_spec.rb Updates warning expectations now that native transport supports span links.
spec/datadog/tracing/transport/native/tracer_span_spec.rb Adds native conversion behavior tests for span-link normalization, UTF-8 rejection, and snapshot cleanup.
spec/datadog/tracing/transport/native/conformance_spec.rb Adds on-the-wire conformance checks for span-link payload content and ordering.
spec/datadog/tracing/distributed/propagation_spec.rb Extends propagation coverage to assert SpanLink#to_hash output for extracted context.
spec/datadog/opentelemetry_spec.rb Adds explicit assertions for OpenTelemetry link translation into canonical to_hash payloads.
lib/datadog/tracing/transport/native.rb Removes “unsupported span links” warning path and updates commentary to reflect new support.
ext/libdatadog_api/trace_exporter.c Implements C-side snapshotting of canonical span-link hashes and exports them via libdatadog setters.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread spec/datadog/tracing/transport/native/transport_spec.rb
Comment thread ext/libdatadog_api/trace_exporter.c
lloeki added 2 commits August 5, 2026 09:52
Keep libdatadog UTF-8 validation consistent with native meta handling,
but make the resulting whole-batch failure explicit. Cover the transport
response and clarify which canonical link fields the wire format supports.
Avoid rb_protect, snapshot allocation, and the libdatadog setter for spans
without links. Empty links are the common path and require no ownership or
validation work before the Rust span is populated.
hoolioh added a commit to DataDog/libdatadog that referenced this pull request Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial
AI assistance and manually reviewed and validated.

# What does this PR do?

Adds owned span-event construction and atomic attachment through the
data-pipeline FFI. It supports typed scalar attributes, homogeneous
typed arrays, empty arrays, event ordering, and UTF-8 validation.

# Motivation

The Rust trace model supports span events, but native exporter consumers
cannot populate them. This fills the span-event portion of the payload
parity work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# Additional Notes

**Stack position: L3, current PR: [DataDog/libdatadog PR
2301](#2301

These open L0-L5 changes are not included in [libdatadog
v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0).
They are planned to ship together in a single libdatadog v39 release,
with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39
release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR
2302](#2302) | Release
prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR
2303](#2303) | [R4
DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR
2305](#2305) | [R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129) |
| **L3 (current)** | **[DataDog/libdatadog PR
2301](#2301 | **[R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134 |
| L4 | [DataDog/libdatadog PR
2304](#2304) | [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR
2300](#2300) | [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in
the listed order is still recommended because it reduces integration and
release-branch conflicts, and lands panic containment before the new
payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130), [R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129), [R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134), [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132), and [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133). All depend
through the common v39 uptake in [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143). The downstream
review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also
depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR
65](DataDog/libdatadog-rb#65) and
[DataDog/libdatadog-rb v38 PR
66](DataDog/libdatadog-rb#66) are separate
ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR
6131](DataDog/dd-trace-rb#6131) and its
`ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or
payload APIs. R3 is stacked on R2, but R3, rather than either
prerequisite package, is the common v39 payload-API uptake.

L0 tracks
[APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5
implement the payload work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop --command cargo test -p libdd-data-pipeline-ffi` passed
58 tests, including scalar values, homogeneous typed arrays, empty
arrays, ordering, ownership, and atomic failure coverage.
- `nix develop --command cargo clippy -p libdd-data-pipeline-ffi
--all-targets -- -D warnings` passed.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- `nix develop --command env
LIBDATADOG_VENDOR_OVERRIDE=/tmp/opencode/native-span-events-ruby bundle
exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb
spec/datadog/tracing/transport/native/send_traces_spec.rb
spec/datadog/tracing/transport/native/transport_spec.rb
spec/datadog/tracing/transport/traces_spec.rb
spec/datadog/tracing/span_event_spec.rb
spec/datadog/tracing/span_operation_spec.rb` passed 123 native and
producer examples, including native wire output and legacy JSON
fallback.
- Targeted Standard and Steep checks for the native transport and
span-event paths passed.




[APMSP-3219]:
https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]:
https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Julio <julio.gonzalez@datadoghq.com>
Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com>
Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
hoolioh added a commit to DataDog/libdatadog that referenced this pull request Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial
AI assistance and manually reviewed and validated.

# What does this PR do?

Adds an atomic batched FFI for complete span links, including 128-bit
trace IDs, span IDs, flags, tracestate, attributes, and ordering. All
borrowed input is validated and copied before insertion.

# Motivation

The Rust trace model supports span links, but native exporter consumers
cannot populate it. This fills the span-link portion of the payload
parity work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# Additional Notes

`dropped_attributes_count` is intentionally deferred by additive commit
[961044e](961044e).
Neither the Agent v0.4 nor indexed V1 span-link schema defines the
field, and current Datadog tracers do not produce non-zero counts. The
original implementation remains in the preceding commit and can be
restored by reverting the deferral commit.

**Stack position: L2, current PR: [DataDog/libdatadog PR
2305](#2305

These open L0-L5 changes are not included in [libdatadog
v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0).
They are planned to ship together in a single libdatadog v39 release,
with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39
release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR
2302](#2302) | Release
prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR
2303](#2303) | [R4
DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130) |
| **L2 (current)** | **[DataDog/libdatadog PR
2305](#2305 | **[R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129 |
| L3 | [DataDog/libdatadog PR
2301](#2301) | [R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR
2304](#2304) | [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132) |
| L5 | [DataDog/libdatadog PR
2300](#2300) | [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in
the listed order is still recommended because it reduces integration and
release-branch conflicts, and lands panic containment before the new
payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR
6130](DataDog/dd-trace-rb#6130), [R5
DataDog/dd-trace-rb PR
6129](DataDog/dd-trace-rb#6129), [R6
DataDog/dd-trace-rb PR
6134](DataDog/dd-trace-rb#6134), [R7
DataDog/dd-trace-rb PR
6132](DataDog/dd-trace-rb#6132), and [R8
DataDog/dd-trace-rb PR
6133](DataDog/dd-trace-rb#6133). All depend
through the common v39 uptake in [R3 DataDog/dd-trace-rb PR
6143](DataDog/dd-trace-rb#6143). The downstream
review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also
depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR
65](DataDog/libdatadog-rb#65) and
[DataDog/libdatadog-rb v38 PR
66](DataDog/libdatadog-rb#66) are separate
ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR
6131](DataDog/dd-trace-rb#6131) and its
`ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or
payload APIs. R3 is stacked on R2, but R3, rather than either
prerequisite package, is the common v39 payload-API uptake.

L0 tracks
[APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5
implement the payload work tracked by
[APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo nextest run -p libdd-data-pipeline-ffi` passed
59 tests, including atomic invalid-UTF-8 failure, ordering, and
ownership coverage.
- `nix develop -c cargo test -p libdd-trace-utils` passed the unit,
integration, and doc-test coverage for all affected encoders and
decoders.
- `nix develop -c cargo clippy -p libdd-trace-utils -p
libdd-data-pipeline-ffi --all-targets -- -D warnings` passed.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- `nix develop --command bundle exec rspec
spec/datadog/tracing/transport/native/tracer_span_spec.rb
spec/datadog/tracing/transport/native/transport_spec.rb
spec/datadog/tracing/transport/native/conformance_spec.rb` passed 119
native transport examples against the locally built artifact.
- `nix develop --command bundle exec rspec
spec/datadog/tracing/span_link_spec.rb
spec/datadog/tracing/distributed/propagation_spec.rb` passed 69
distributed-propagation examples with 1 existing pending example.
- `nix develop --command env
BUNDLE_GEMFILE=gemfiles/ruby_4.0_opentelemetry.gemfile bundle exec rspec
spec/datadog/opentelemetry_spec.rb` passed 164 OpenTelemetry examples.
- Full Standard and Steep checks passed on the matching consumer branch.




[APMSP-3219]:
https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]:
https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com>
Co-authored-by: Julio <julio.gonzalez@datadoghq.com>
Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
Carry the published v39 package and generated lockfiles into the native
span-link consumer without rewriting its reviewed commits.
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 6, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Introduces a validated, language-neutral structured-value token ABI and standalone MessagePack encoder. It supports nil, booleans, signed and unsigned integers, floats, UTF-8 strings, binary strings, arrays, and maps, with depth and shape validation.

# Motivation

The opaque blob API in [DataDog/libdatadog PR 2303](#2303) restores transitional parity but still requires every consumer to encode MessagePack. This stage moves validated structured-value encoding into libdatadog as part of [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# Additional Notes

**Stack position: L4, current PR: [DataDog/libdatadog PR 2304](#2304), stacked on L1 [DataDog/libdatadog PR 2303](#2303

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| **L4 (current)** | **[DataDog/libdatadog PR 2304](#2304 | **[R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132 |
| L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

This PR remains a draft because its review and final rebase depend on L1. Its matching consumer [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) is also a draft pending this API, the v39 package, and [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). L5 remains draft until this PR is ready.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop -c cargo test -p libdd-data-pipeline-ffi structured_value` passed structured-value encoding, ownership, malformed-stream, shape, and depth-limit tests.
- `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed the complete data-pipeline FFI suite.
- `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed.
- `nix develop .#nightly -c cargo fmt --all -- --check` passed.
- The release artifact built successfully with `nix develop -c cargo run -p builder --bin release -- --out /tmp/opencode/libdatadog-m2/x86_64-linux`.
- The matching consumer's native suite passed 122 examples, AppSec passed 807 examples, and AI Guard passed 125 examples against the local artifact.
- Full Standard and Steep checks passed on the matching consumer branch.




[APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Aug 6, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

# What does this PR do?

Adds direct structured-value insertion into `TracerSpan.meta_struct`, reusing the validator and encoder from [DataDog/libdatadog PR 2304](#2304) while moving the encoded allocation directly into the span. The opaque blob and standalone encoder APIs remain available.

# Motivation

The standalone encoder still requires consumers to manage an intermediate encoded blob and an additional value copy. The direct setter provides the final efficient API for structured payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# Additional Notes

**Stack position: L5, current PR: [DataDog/libdatadog PR 2300](#2300), stacked on L4 [DataDog/libdatadog PR 2304](#2304

These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143).

Complete expected libdatadog review and merge order for that single v39 release:

| Level | libdatadog PR | Matching dd-trace-rb consumer |
| --- | --- | --- |
| L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer |
| L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) |
| L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) |
| L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) |
| L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) |
| **L5 (current)** | **[DataDog/libdatadog PR 2300](#2300 | **[R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133 |

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.

[DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.

This PR remains a draft because its review and final rebase depend on L4 and, transitively, L1. Its matching consumer [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) is also a draft pending this API, the v39 package, and [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). L4 remains draft until L1 is ready.

L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219).

# How to test the change?

Prior local validation completed successfully:

- `nix develop --command cargo test -p libdd-data-pipeline-ffi --locked` passed 68 tests, including direct-setter/standalone-encoder equivalence, atomic failure, invalid handles, overwrite behaviour, and ownership.
- `nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets --locked -- -D warnings` passed.
- `nix develop .#nightly --command cargo fmt --all -- --check` passed.
- The release artifact built successfully with `nix develop -c cargo run --bin release -- --out /tmp/opencode/libdatadog-m3/x86_64-linux`.
- The matching consumer's native suite passed 122 examples, and AppSec passed 807 examples against the local artifact.
- `nix develop .#ruby40 --command nix shell nixpkgs#valgrind --command bundle exec ruby_memcheck vendor/bundle/ruby/4.0.0/gems/rspec-core-3.13.6/exe/rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/trace_exporter_spec.rb` passed with no Valgrind memory errors.
- Full Standard and Steep checks passed on the matching consumer branch.




[APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Co-authored-by: ekump <edmund.kump@datadoghq.com>
Base automatically changed from lloeki/native-libdatadog-payload-api-uptake to master August 7, 2026 10:13
lloeki added 2 commits August 7, 2026 13:47
The span-links snapshot path normalizes nullable strings before conversion, leaving the old helper unused under Clang warnings-as-errors.
@pr-commenter

pr-commenter Bot commented Aug 7, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-10 11:41:50

Comparing candidate commit 81dd627 in PR branch lloeki/native-span-links-consumer with baseline commit c674c96 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:tracing - trace.to_digest - Continue

  • unstable throughput [-1210.358op/s; +1890.023op/s] or [-4.149%; +6.478%]

# Conflicts:
#	ext/libdatadog_api/trace_exporter.c
Comment thread ext/libdatadog_api/trace_exporter.c Outdated
The @links type check runs after prepare_metastruct has allocated
C-owned token storage. The caller's free_raw_span ensure handler only
releases owner->span, so raising there leaked the prepared meta_struct.
Every other raise path after prepare_metastruct already frees it.

Unreachable from a real Span, but measurable when forced: 20k rejected
conversions of a span with a non-empty @Metastruct grew RSS by 2043MB
before, 0MB after.
@p-datadog

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: adab38d60f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants