feat(data-pipeline): set structured span values - #2300
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 15 commits intoAug 6, 2026
Conversation
Expose a C setter that copies an opaque binary value into a span's `meta_struct` map. Native tracers can now supply pre-encoded MessagePack without transferring ownership or requiring Rust to interpret it. Reject null span handles and non-UTF-8 keys. Reusing a key replaces its value, and the copied bytes remain owned by the span after the call.
Expose a flat preorder token ABI that encodes nil, scalar, string, binary, array, and map values into an owned MessagePack blob. This lets C callers build structured metadata without implementing MessagePack or sharing the lifetime of their input buffers. Reject malformed token streams, invalid kinds, booleans, UTF-8, counts, and nesting deeper than 64 levels. Successful output contains exactly one value and remains valid until explicitly freed.
Exercise the generated C ABI from the packaged trace exporter example. Mutate the caller's source buffers after encoding and compare the blob with fixed MessagePack bytes to prove that output ownership is independent of the input lifetime. Mirror the ownership check in Rust and document that token byte slices are borrowed only for the synchronous encoding call.
Add a span setter that accepts structured-value tokens and encodes them directly into `meta_struct`. Native tracers can populate structured metadata without managing an intermediate encoded blob or paying an additional value copy. Validate the key and complete token stream before replacing an existing entry. Encoding failures and invalid handles therefore leave the span unchanged.
BenchmarksComparisonCandidateCandidate benchmark detailsBaselineBaseline benchmark details |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 61c202b | Docs | Datadog PR Page | Give us feedback! |
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Member
Author
|
Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6133. |
lloeki
force-pushed
the
lloeki/native-meta-struct-m2
branch
from
July 30, 2026 12:03
c402ab6 to
1c4db0d
Compare
lloeki
force-pushed
the
lloeki/native-meta-struct-m3
branch
from
July 30, 2026 12:03
d8dad0b to
bad622f
Compare
This was referenced Jul 30, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Ensures the combined libdatadog profiling artifact enables `catch_panic` for data-pipeline FFI entry points. It also adds a packaged C regression that triggers capacity overflow and verifies that the FFI returns an error instead of aborting the process. # Motivation The standalone data-pipeline FFI crate enables panic containment by default, but the combined profiling artifact disables dependency defaults and did not re-enable it. This closes [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830) before the native trace exporter payload APIs ship together. # Additional Notes **Stack position: L0, current PR: [DataDog/libdatadog PR 2302](#2302 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | **L0 (current)** | **[DataDog/libdatadog PR 2302](#2302 | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-profiling-ffi --features data-pipeline-ffi` passed the combined profiling/data-pipeline FFI suite: 57 tests. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic trace_chunks_new_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo test -p libdd-data-pipeline-ffi --features catch_panic begin_chunk_with_overflow_capacity_returns_panic_error` passed. - `nix develop -c cargo ffi-test --filter trace_exporter_panic_containment` built and passed the packaged C panic-containment regression. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop -c cargo clippy -p libdd-profiling-ffi --all-targets --features data-pipeline-ffi -- -D warnings` passed. [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: hoolioh <julio.gonzalez@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 4, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an opaque per-key `meta_struct` blob setter to `TracerSpan`. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model. # Motivation Native exporter consumers cannot currently populate `meta_struct`, which blocks payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). The opaque setter provides the safe first stage before the language-neutral structured-value APIs. # Additional Notes **Stack position: L1, current PR: [DataDog/libdatadog PR 2303](#2303 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | **L1 (current)** | **[DataDog/libdatadog PR 2303](#2303 | **[R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130 | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys. - `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly -c cargo fmt --all -- --check` passed. - The generated C header was rebuilt through `nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux` and consumed successfully by the matching dd-trace-rb branch. - `nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb` passed 116 examples, including AppSec and AI Guard-shaped wire conformance. - Targeted Standard and Steep checks for the native transport passed. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
hoolioh
added a commit
that referenced
this pull request
Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds owned span-event construction and atomic attachment through the data-pipeline FFI. It supports typed scalar attributes, homogeneous typed arrays, empty arrays, event ordering, and UTF-8 validation. # Motivation The Rust trace model supports span events, but native exporter consumers cannot populate them. This fills the span-event portion of the payload parity work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes **Stack position: L3, current PR: [DataDog/libdatadog PR 2301](#2301 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | **L3 (current)** | **[DataDog/libdatadog PR 2301](#2301 | **[R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134 | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop --command cargo test -p libdd-data-pipeline-ffi` passed 58 tests, including scalar values, homogeneous typed arrays, empty arrays, ordering, ownership, and atomic failure coverage. - `nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop --command env LIBDATADOG_VENDOR_OVERRIDE=/tmp/opencode/native-span-events-ruby bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/traces_spec.rb spec/datadog/tracing/span_event_spec.rb spec/datadog/tracing/span_operation_spec.rb` passed 123 native and producer examples, including native wire output and legacy JSON fallback. - Targeted Standard and Steep checks for the native transport and span-event paths passed. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Julio <julio.gonzalez@datadoghq.com> Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com> Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
hoolioh
added a commit
that referenced
this pull request
Aug 5, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an atomic batched FFI for complete span links, including 128-bit trace IDs, span IDs, flags, tracestate, attributes, and ordering. All borrowed input is validated and copied before insertion. # Motivation The Rust trace model supports span links, but native exporter consumers cannot populate it. This fills the span-link portion of the payload parity work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes `dropped_attributes_count` is intentionally deferred by additive commit [961044e](961044e). Neither the Agent v0.4 nor indexed V1 span-link schema defines the field, and current Datadog tracers do not produce non-zero counts. The original implementation remains in the preceding commit and can be restored by reverting the deferral commit. **Stack position: L2, current PR: [DataDog/libdatadog PR 2305](#2305 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | **L2 (current)** | **[DataDog/libdatadog PR 2305](#2305 | **[R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129 | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo nextest run -p libdd-data-pipeline-ffi` passed 59 tests, including atomic invalid-UTF-8 failure, ordering, and ownership coverage. - `nix develop -c cargo test -p libdd-trace-utils` passed the unit, integration, and doc-test coverage for all affected encoders and decoders. - `nix develop -c cargo clippy -p libdd-trace-utils -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly --command cargo fmt --all -- --check` passed. - `nix develop --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb` passed 119 native transport examples against the locally built artifact. - `nix develop --command bundle exec rspec spec/datadog/tracing/span_link_spec.rb spec/datadog/tracing/distributed/propagation_spec.rb` passed 69 distributed-propagation examples with 1 existing pending example. - `nix develop --command env BUNDLE_GEMFILE=gemfiles/ruby_4.0_opentelemetry.gemfile bundle exec rspec spec/datadog/opentelemetry_spec.rb` passed 164 OpenTelemetry examples. - Full Standard and Steep checks passed on the matching consumer branch. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Edmund Kump <edmund.kump@datadoghq.com> Co-authored-by: Julio <julio.gonzalez@datadoghq.com> Co-authored-by: Julio Gonzalez <107922352+hoolioh@users.noreply.github.com>
ekump
requested review from
btthomas,
sameerank and
vpellan
and removed request for
a team
August 5, 2026 22:29
![datadog-prod-us1-5[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The direct structured-span setter reuses the existing fully validating MessagePack encoder, validates the key and complete token stream before mutating the span, and preserves the established VecMap overwrite semantics. No diff-attributable behavioral regression was identified; package execution was blocked by the sandbox's unavailable patched proptest Git dependency.
🤖 Datadog Autotest · Commit 8591fc8 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
ekump
force-pushed
the
lloeki/native-meta-struct-m3
branch
from
August 5, 2026 23:21
8591fc8 to
bad622f
Compare
ekump
approved these changes
Aug 5, 2026
Bring the rmp-backed M2 encoder and current data-pipeline FFI APIs into the direct structured span setter without rewriting its reviewed commit. # Conflicts: # libdd-data-pipeline-ffi/src/structured_value.rs # libdd-data-pipeline-ffi/src/tracer.rs
Define token-kind values once as public `u8` constants, derive an internal Rust enum for readable dispatch, and let cbindgen emit the existing C names. Keep the FFI field raw so unknown caller values remain safely rejectable.
Carry the safe generated token constants into the direct structured setter without rewriting its existing merge resolution.
Apply the repository-pinned nightly rustfmt layout to the token-kind match guards.
Carry the pinned rustfmt output from the structured encoder base into the direct setter branch.
hoolioh
approved these changes
Aug 6, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 6, 2026
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Introduces a validated, language-neutral structured-value token ABI and standalone MessagePack encoder. It supports nil, booleans, signed and unsigned integers, floats, UTF-8 strings, binary strings, arrays, and maps, with depth and shape validation. # Motivation The opaque blob API in [DataDog/libdatadog PR 2303](#2303) restores transitional parity but still requires every consumer to encode MessagePack. This stage moves validated structured-value encoding into libdatadog as part of [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # Additional Notes **Stack position: L4, current PR: [DataDog/libdatadog PR 2304](#2304), stacked on L1 [DataDog/libdatadog PR 2303](#2303 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | L1 | [DataDog/libdatadog PR 2303](#2303) | [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130) | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | **L4 (current)** | **[DataDog/libdatadog PR 2304](#2304 | **[R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132 | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. This PR remains a draft because its review and final rebase depend on L1. Its matching consumer [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) is also a draft pending this API, the v39 package, and [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). L5 remains draft until this PR is ready. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-data-pipeline-ffi structured_value` passed structured-value encoding, ownership, malformed-stream, shape, and depth-limit tests. - `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed the complete data-pipeline FFI suite. - `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly -c cargo fmt --all -- --check` passed. - The release artifact built successfully with `nix develop -c cargo run -p builder --bin release -- --out /tmp/opencode/libdatadog-m2/x86_64-linux`. - The matching consumer's native suite passed 122 examples, AppSec passed 807 examples, and AI Guard passed 125 examples against the local artifact. - Full Standard and Steep checks passed on the matching consumer branch. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
There was a problem hiding this comment.
More details
The structured-value encoder and direct span setter handle the examined scalar, nested-container, malformed-input, ownership, and atomic-update cases without exposing a clear behavioral regression. Full package execution was unavailable because Cargo could not fetch the repository's pinned proptest git revision in the sandbox.
🤖 Datadog Autotest · Commit 776e5af · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Reconcile the merged structured-value encoder with the direct span setter while preserving the shared validation helper.
gh-worker-ownership-write-b05516
Bot
removed the request for review
from a team
August 6, 2026 13:55
Collaborator
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: queued
mergequeue-status: in_progress
and removed
mergequeue-status: queued
labels
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
lloeki/native-meta-struct-m3
branch
August 6, 2026 15:24
bwoebi
added a commit
that referenced
this pull request
Aug 7, 2026
…mote_config/agentless_fetcher * 'main' of github.com:DataDog/libdatadog: feat(data-pipeline)!: add flush_and_close to the trace buffer (#2313) refactor(rc)!: make conversion from RemoteConfigProduct back and forth generally available (#2325) chore: release v40.0.0 (#2326) feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery (#2117) feat(data-pipeline): set structured span values (#2300) feat(telemetry)!: Add Installation signature and AppProduct changes payloads (#2213) fix(libdd-trace-utils): apply SpanLink flags masking when v0.5 json encoding (#2314) feat(data-pipeline): encode structured values (#2304) feat(crashtracking): send debug log when no data is received at all (#2321) chore: release v39.0.0 (#2324) feat(data-pipeline): add span links FFI (#2305) fix(crashtracking): check fields and exclude uuid for `has_data` (#2322) feat(data-pipeline): add span events FFI (#2301)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.
What does this PR do?
Adds direct structured-value insertion into
TracerSpan.meta_struct, reusing the validator and encoder from DataDog/libdatadog PR 2304 while moving the encoded allocation directly into the span. The opaque blob and standalone encoder APIs remain available.Motivation
The standalone encoder still requires consumers to manage an intermediate encoded blob and an additional value copy. The direct setter provides the final efficient API for structured payload parity tracked by APMSP-3219.
Additional Notes
Stack position: L5, current PR: DataDog/libdatadog PR 2300, stacked on L4 DataDog/libdatadog PR 2304.
These open L0-L5 changes are not included in libdatadog v38.0.0. They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by R3 DataDog/dd-trace-rb PR 6143.
Complete expected libdatadog review and merge order for that single v39 release:
L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.
The matching payload consumers are R4 DataDog/dd-trace-rb PR 6130, R5 DataDog/dd-trace-rb PR 6129, R6 DataDog/dd-trace-rb PR 6134, R7 DataDog/dd-trace-rb PR 6132, and R8 DataDog/dd-trace-rb PR 6133. All depend through the common v39 uptake in R3 DataDog/dd-trace-rb PR 6143. The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.
DataDog/libdatadog-rb v37 PR 65 and DataDog/libdatadog-rb v38 PR 66 are separate ordered prerequisite releases only for R2 DataDog/dd-trace-rb PR 6131 and its
ForkSafeRuntimeuptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.This PR remains a draft because its review and final rebase depend on L4 and, transitively, L1. Its matching consumer R8 DataDog/dd-trace-rb PR 6133 is also a draft pending this API, the v39 package, and R3 DataDog/dd-trace-rb PR 6143. L4 remains draft until L1 is ready.
L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.
How to test the change?
Prior local validation completed successfully:
nix develop --command cargo test -p libdd-data-pipeline-ffi --lockedpassed 68 tests, including direct-setter/standalone-encoder equivalence, atomic failure, invalid handles, overwrite behaviour, and ownership.nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets --locked -- -D warningspassed.nix develop .#nightly --command cargo fmt --all -- --checkpassed.nix develop -c cargo run --bin release -- --out /tmp/opencode/libdatadog-m3/x86_64-linux.nix develop .#ruby40 --command nix shell nixpkgs#valgrind --command bundle exec ruby_memcheck vendor/bundle/ruby/4.0.0/gems/rspec-core-3.13.6/exe/rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/trace_exporter_spec.rbpassed with no Valgrind memory errors.