Skip to content

feat(wall): support signal suppression for unfiltered threads - #663

Open
kaahos wants to merge 27 commits into
mainfrom
paul.fournillon/wallclock-all-threads
Open

feat(wall): support signal suppression for unfiltered threads#663
kaahos wants to merge 27 commits into
mainfrom
paul.fournillon/wallclock-all-threads

Conversation

@kaahos

@kaahos kaahos commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?:

Extends wall-clock signal suppression to threads sampled through the existing explicit filter= mode.

When wall profiling runs with both filter= and wallprecheck=true, the profiler now:

  • Tracks thread identity and lifecycle independently from context-filter membership.
  • Registers Java threads that already exist when profiling starts.
  • Registers new threads through the existing thread lifecycle callbacks.
  • Allows block and park hooks to lazily associate pre-existing threads with their registry slots.
  • Makes the thread metadata available to both the ASGCT and JVMTI wall-clock paths.
  • Suppresses repeated samples only for profiler-owned blocking runs that remain outside the context window.
  • Validates thread identity, slot lifecycle, and context transitions before applying suppression.

The existing context-filtered behavior remains unchanged. This PR does not change the meaning of filter=.

Motivation:

With filter=, wall-clock profiling already samples every thread. However, threads outside an active context did not have the lifecycle metadata required to recognize and suppress repeated samples from the same profiler-owned blocking run.

This change closes that metadata gap. It also provides the thread lifecycle machinery needed by the follow-up TaskBlock work for non-context threads.

Implementation notes:

  • Thread identity is indexed by native TID while context-window membership remains separate.
  • Lifecycle generations prevent reused slots or TIDs from inheriting stale state.
  • Context-window epochs prevent suppression if a thread enters or leaves a context during a blocking run.
  • Registry tracking is activated only for wall recordings using explicit filter= together with wallprecheck=true.
  • Tracking is disabled again when wall profiling is absent, precheck is disabled, wall startup fails, or a later recording uses another configuration.
  • Both ASGCT and JVMTI wall-clock collection paths use the same registry validation.

How to test the change?:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug_threadFilter_ut :ddprof-lib:gtestDebug_wallprecheck_args_ut
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests="*UnfilteredWallPrecheck*"
  • ./.claude/commands/build-and-summarize testDebug

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials
    of any kind, I've requested a security review.
  • This PR doesn't touch any of that.
  • JIRA: [JIRA-XXXX]

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #32491259439 | Commit: 96c1a29 | Duration: 15m 9s (longest job)

1 of 32 test jobs failed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Failed Tests

musl-amd64/debug / 17-librca

Job: View logs

No detailed failure information available. Check the job logs.

Summary: Total: 32 | Passed: 31 | Failed: 1


Updated: 2026-08-21 14:34:40 UTC

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 3 times, most recently from 3902a14 to f302dd5 Compare July 19, 2026 12:00
@kaahos kaahos changed the title feat(wall): add configurable wall thread scope feat(wall): support signal suppression for unfiltered threads Jul 19, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 2 times, most recently from 0ad425c to fb1c44d Compare July 19, 2026 18:43
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit fb1c44d)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125547869 Commit: fb1c44de77197780eae60267ff98bd38a907a558

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.9% (2679→2836 ms)
  • 🔴 future-genetic (JDK 21): runtime +3.8% (2083→2162 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10210 ms (21 iters) ✅ 10256 ms (21 iters) ≈ +0.5% (±10.9%) — / —
akka-uct 25 ✅ 8953 ms (24 iters) ✅ 8890 ms (24 iters) ≈ -0.7% (±10.2%) — / —
finagle-chirper 21 ✅ 6000 ms (33 iters) ✅ 5960 ms (33 iters) ≈ -0.7% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5431 ms (36 iters) ✅ 5443 ms (36 iters) ≈ +0.2% (±24.6%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2679 ms (70 iters) ✅ 2836 ms (66 iters) 🔴 +5.9% — / —
fj-kmeans 25 ✅ 2847 ms (66 iters) ✅ 2828 ms (66 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 21 ✅ 2083 ms (89 iters) ✅ 2162 ms (86 iters) 🔴 +3.8% — / —
future-genetic 25 ✅ 2076 ms (90 iters) ✅ 2122 ms (88 iters) ≈ +2.2% (±2.5%) — / —
naive-bayes 21 ✅ 1252 ms (136 iters) ✅ 1248 ms (137 iters) ≈ -0.3% (±32.7%) — / —
naive-bayes 25 ✅ 1008 ms (170 iters) ✅ 1023 ms (167 iters) ≈ +1.5% (±31.7%) — / —
reactors 21 ✅ 16312 ms (15 iters) ✅ 16357 ms (15 iters) ≈ +0.3% (±8.8%) — / —
reactors 25 ✅ 18416 ms (15 iters) ✅ 18780 ms (15 iters) ≈ +2% (±3.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1891 / 2004 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2305 / 2283 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 2 8183 / 8674 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8087 / 8641 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1270 / 1278 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1282 / 1275 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2966 / 3007 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2992 / 2972 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 6 3491 / 3501 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 5 3488 / 3493 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1711 / 1786 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1819 / 1920 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from fb1c44d to 1118f62 Compare July 19, 2026 21:16
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 1118f62)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125554643 Commit: 1118f62831a4384bdf0283ef01a626c5bdc48af5

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -3.9% (2808→2698 ms)
  • 🟢 future-genetic (JDK 25): runtime -4.2% (2088→2001 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10294 ms (21 iters) ✅ 10251 ms (21 iters) ≈ -0.4% (±11.4%) — / —
akka-uct 25 ✅ 8933 ms (24 iters) ✅ 8835 ms (24 iters) ≈ -1.1% (±10%) — / —
finagle-chirper 21 ✅ 6016 ms (33 iters) ✅ 6034 ms (33 iters) ≈ +0.3% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5462 ms (36 iters) ✅ 5452 ms (36 iters) ≈ -0.2% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2808 ms (66 iters) ✅ 2698 ms (70 iters) 🟢 -3.9% — / —
fj-kmeans 25 ✅ 2812 ms (66 iters) ✅ 2813 ms (66 iters) ≈ +0% (±2.6%) — / —
future-genetic 21 ✅ 2067 ms (90 iters) ✅ 2083 ms (90 iters) ≈ +0.8% (±2.7%) — / —
future-genetic 25 ✅ 2088 ms (89 iters) ✅ 2001 ms (93 iters) 🟢 -4.2% — / —
naive-bayes 21 ✅ 1239 ms (138 iters) ✅ 1231 ms (138 iters) ≈ -0.6% (±32.7%) — / —
naive-bayes 25 ✅ 1019 ms (168 iters) ✅ 1011 ms (169 iters) ≈ -0.8% (±32.1%) — / —
reactors 21 ✅ 16149 ms (15 iters) ✅ 16215 ms (15 iters) ≈ +0.4% (±8.4%) — / —
reactors 25 ✅ 18598 ms (15 iters) ✅ 18345 ms (15 iters) ≈ -1.4% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1978 / 1973 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2287 / 2449 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8978 / 8724 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8633 / 8297 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1257 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1280 / 1268 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 3 2909 / 2923 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 5 2847 / 2887 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 4 3492 / 3493 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 2 3463 / 3473 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1761 / 1721 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1916 / 1880 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 40dbe8e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125596529 Commit: 40dbe8e833888192b6026558590bce1cf0ad1c9f

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.7% (2823→2748 ms)
  • 🔴 future-genetic (JDK 21): runtime +4.1% (2041→2125 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (1994→2110 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10340 ms (21 iters) ✅ 10312 ms (21 iters) ≈ -0.3% (±11.6%) — / —
akka-uct 25 ✅ 8871 ms (24 iters) ✅ 8849 ms (24 iters) ≈ -0.2% (±10.1%) — / —
finagle-chirper 21 ✅ 6017 ms (33 iters) ✅ 6015 ms (33 iters) ≈ -0% (±25%) ⚠️ W:3 / ⚠️ W:4
finagle-chirper 25 ✅ 5470 ms (36 iters) ✅ 5509 ms (36 iters) ≈ +0.7% (±24.2%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2823 ms (66 iters) ✅ 2748 ms (68 iters) 🟢 -2.7% — / —
fj-kmeans 25 ✅ 2856 ms (66 iters) ✅ 2847 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2041 ms (90 iters) ✅ 2125 ms (88 iters) 🔴 +4.1% — / —
future-genetic 25 ✅ 1994 ms (93 iters) ✅ 2110 ms (88 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1247 ms (137 iters) ✅ 1259 ms (135 iters) ≈ +1% (±33.5%) — / —
naive-bayes 25 ✅ 1027 ms (167 iters) ✅ 1018 ms (168 iters) ≈ -0.9% (±31.5%) — / —
reactors 21 ✅ 16583 ms (15 iters) ✅ 16798 ms (15 iters) ≈ +1.3% (±7.5%) — / —
reactors 25 ✅ 18582 ms (15 iters) ✅ 18610 ms (15 iters) ≈ +0.2% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 6 1978 / 1998 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2344 / 2271 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8863 / 8312 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8368 / 8738 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 7 / 1 1284 / 1242 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1302 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2979 / 2980 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 2847 / 2862 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 5 3520 / 3534 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 3 3523 / 3509 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 2 1682 / 1809 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1806 / 1871 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings July 21, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Extends wall-clock “owned-block” signal suppression to unfiltered (explicit filter=) wall recordings when wallprecheck=true, by introducing a thread registry keyed by native TID and integrating it into both ASGCT and JVMTI wall-clock sampling paths.

Changes:

  • Added unfiltered wall thread registry support (native TID indexing, lifecycle generations, recording epochs) and wired it into profiler lifecycle + wall-clock precheck logic.
  • Added a bounded candidate-selection/backfill mechanism for precheck to avoid reverting to O(N) registry work per wall tick under heavy suppression.
  • Added Java integration tests, C++ unit tests, and a JMH benchmark to validate behavior and measure overhead.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java New integration coverage for suppression behavior in explicit-empty-filter wall recordings.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java Verifies registry activation does not leak across recording restarts and handles stopped gaps safely.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java Ensures the unfiltered path works when wall sampling delegates stacks via JVMTI.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java Confirms J9 wall engine capability gating prevents activation of unfiltered tracking.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java Adds beforeProfilerStart() hook to allow test setup before profiler thread callbacks are enabled.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java Exposes owned-block hooks to benchmarks without widening core API surface.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java Adds throughput benchmark comparing wallprecheck=false/true as owned-block population grows.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp Adds capability tests and verifies explicit-empty vs omitted filter parsing behavior.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp Unit-tests the new bounded candidate selection logic.
ddprof-lib/src/test/cpp/threadFilter_ut.cpp Expands tests for registry/epoch/lifecycle behavior and adjusts TID usage in recovery test.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h Adds reusable “randomized prefix without replacement” candidate selection with stats.
ddprof-lib/src/main/cpp/wallClock.h Integrates candidate selector and adds bounded-visit backfill path to the wall timer loop.
ddprof-lib/src/main/cpp/wallClock.cpp Updates precheck suppression to validate registry identity + context transitions; supports lazy registry lookups.
ddprof-lib/src/main/cpp/threadFilter.h Introduces registry state (TID index, epochs, lifecycle generation) and APIs for unfiltered tracking.
ddprof-lib/src/main/cpp/threadFilter.cpp Implements lock-free lookups + mutex-serialized writers for TID indexing, epoch refresh, and retirement.
ddprof-lib/src/main/cpp/profiler.h Declares bootstrap of existing Java threads for registry population.
ddprof-lib/src/main/cpp/profiler.cpp Boots existing Java threads into registry when unfiltered tracking is active; deactivates registry on stop/failure.
ddprof-lib/src/main/cpp/jvmThread.h Adds capability check for cross-thread native TID lookup support.
ddprof-lib/src/main/cpp/jvmThread.cpp Implements native TID lookup support predicate.
ddprof-lib/src/main/cpp/javaApi.cpp Ensures TLS slot binding is validated/refreshed before using owned-block and filter APIs.
ddprof-lib/src/main/cpp/engine.h Adds wall-engine capability hook for unfiltered precheck support.
ddprof-lib/src/main/cpp/counters.h Adds counters for registry bootstrap and bounded-lookup/backfill diagnostics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/wallClockCandidateSelector.h
Comment thread ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp
@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d82291f)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125875202 Commit: d82291ff62852b63091350a05ebf26283fa39029

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10141 ms (21 iters) ✅ 10278 ms (21 iters) ≈ +1.4% (±11.9%) — / —
akka-uct 25 ✅ 8800 ms (24 iters) ✅ 8939 ms (24 iters) ≈ +1.6% (±9.7%) — / —
finagle-chirper 21 ✅ 6003 ms (33 iters) ✅ 6060 ms (33 iters) ≈ +0.9% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5463 ms (36 iters) ✅ 5482 ms (36 iters) ≈ +0.3% (±24.1%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2780 ms (68 iters) ✅ 2771 ms (68 iters) ≈ -0.3% (±2.7%) — / —
fj-kmeans 25 ✅ 2752 ms (68 iters) ✅ 2824 ms (66 iters) ≈ +2.6% (±2.7%) — / —
future-genetic 21 ✅ 2082 ms (90 iters) ✅ 2035 ms (91 iters) ≈ -2.3% (±2.6%) — / —
future-genetic 25 ✅ 2101 ms (88 iters) ✅ 2055 ms (90 iters) ≈ -2.2% (±2.6%) — / —
naive-bayes 21 ✅ 1299 ms (132 iters) ✅ 1232 ms (138 iters) ≈ -5.2% (±31.7%) — / —
naive-bayes 25 ✅ 1011 ms (169 iters) ✅ 1013 ms (169 iters) ≈ +0.2% (±31.4%) — / —
reactors 21 ✅ 16511 ms (15 iters) ✅ 16445 ms (15 iters) ≈ -0.4% (±9.8%) — / —
reactors 25 ✅ 18516 ms (15 iters) ✅ 18146 ms (15 iters) ≈ -2% (±6.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1925 / 1914 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 3 2117 / 2331 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8401 / 8842 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8557 / 8073 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 3 1243 / 1272 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1265 / 1283 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 1 3054 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2890 / 2867 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 3 3522 / 3464 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 5 3452 / 3463 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1679 / 1552 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1933 / 1805 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit ec76f58)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125883187 Commit: ec76f5829d243f516dee70833f3dfbe7a2b9eaa3

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -4.1% (2839→2724 ms)
  • 🔴 future-genetic (JDK 21): runtime +3% (2066→2127 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (2006→2122 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10350 ms (21 iters) ✅ 10305 ms (21 iters) ≈ -0.4% (±11.3%) — / —
akka-uct 25 ✅ 8864 ms (24 iters) ✅ 8976 ms (24 iters) ≈ +1.3% (±10.4%) — / —
finagle-chirper 25 ✅ 5472 ms (36 iters) ✅ 5418 ms (36 iters) ≈ -1% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2839 ms (66 iters) ✅ 2724 ms (68 iters) 🟢 -4.1% — / —
fj-kmeans 25 ✅ 2851 ms (66 iters) ✅ 2836 ms (66 iters) ≈ -0.5% (±2.6%) — / —
future-genetic 21 ✅ 2066 ms (89 iters) ✅ 2127 ms (87 iters) 🔴 +3% — / —
future-genetic 25 ✅ 2006 ms (92 iters) ✅ 2122 ms (87 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1277 ms (134 iters) ✅ 1289 ms (133 iters) ≈ +0.9% (±32.4%) — / —
naive-bayes 25 ✅ 1021 ms (167 iters) ✅ 1023 ms (168 iters) ≈ +0.2% (±31.6%) — / —
reactors 21 ✅ 15903 ms (15 iters) ✅ 16452 ms (15 iters) ≈ +3.5% (±8.8%) — / —
reactors 25 ✅ 18032 ms (15 iters) ✅ 18628 ms (15 iters) ≈ +3.3% (±4.7%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 3 2089 / 2428 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 7 / 2 8520 / 8487 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 4 / 2 8382 / 8319 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / 4 1278 / 1277 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / ✅ 1290 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 2 2953 / 3032 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 2 2846 / 2856 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 3 3534 / 3561 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 3 3480 / 3510 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1614 / 1718 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 3 1962 / 1876 ✅ / ✅ ✅ / ✅

@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 82e05b6)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128529459 Commit: 82e05b6f02f2f64cd9a39e9a2791b4fb2388587e

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 21): runtime +3.9% (2061→2142 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10220 ms (21 iters) ✅ 10147 ms (21 iters) ≈ -0.7% (±11%) — / —
akka-uct 25 ✅ 8860 ms (24 iters) ✅ 8828 ms (24 iters) ≈ -0.4% (±9.6%) — / —
finagle-chirper 21 ✅ 5958 ms (33 iters) ✅ 6066 ms (33 iters) ≈ +1.8% (±25.6%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5534 ms (36 iters) ✅ 5457 ms (36 iters) ≈ -1.4% (±24.1%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2724 ms (68 iters) ✅ 2731 ms (68 iters) ≈ +0.3% (±2.7%) — / —
fj-kmeans 25 ✅ 2850 ms (66 iters) ✅ 2848 ms (66 iters) ≈ -0.1% (±2.5%) — / —
future-genetic 21 ✅ 2061 ms (90 iters) ✅ 2142 ms (86 iters) 🔴 +3.9% — / —
future-genetic 25 ✅ 2019 ms (92 iters) ✅ 2043 ms (91 iters) ≈ +1.2% (±2.6%) — / —
naive-bayes 21 ✅ 1306 ms (132 iters) ✅ 1267 ms (135 iters) ≈ -3% (±32.1%) — / —
naive-bayes 25 ✅ 1004 ms (170 iters) ✅ 1012 ms (169 iters) ≈ +0.8% (±31.7%) — / —
reactors 21 ✅ 15889 ms (15 iters) ✅ 15952 ms (15 iters) ≈ +0.4% (±7.3%) — / —
reactors 25 ✅ 18632 ms (15 iters) ✅ 18335 ms (15 iters) ≈ -1.6% (±4.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 2 1922 / 1896 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2247 / 2407 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 3 8773 / 8354 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8322 / 8456 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 5 / 1 1222 / 1289 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1268 / 1286 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / ✅ 2986 / 2986 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 1 2969 / 2912 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / 2 3528 / 3525 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 1 3507 / 3462 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1594 / 1600 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / 3 1948 / 1838 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings August 3, 2026 10:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit f5b81de)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128545992 Commit: f5b81de3bea33b0718817cf344261bda92848074

⚠️ Significant outliers

  • ⚠️ future-genetic (JDK 25): wallclock unwinds skipped — latest 0 → dev 2874 (new)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10374 ms (21 iters) ✅ 10199 ms (21 iters) ≈ -1.7% (±10.6%) — / —
akka-uct 25 ✅ 8859 ms (24 iters) ✅ 8855 ms (24 iters) ≈ -0% (±9.8%) — / —
finagle-chirper 21 ✅ 6011 ms (33 iters) ✅ 5983 ms (33 iters) ≈ -0.5% (±25.1%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5452 ms (36 iters) ✅ 5461 ms (36 iters) ≈ +0.2% (±23.8%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2770 ms (68 iters) ✅ 2700 ms (70 iters) ≈ -2.5% (±2.7%) — / —
fj-kmeans 25 ✅ 2849 ms (66 iters) ✅ 2855 ms (66 iters) ≈ +0.2% (±2.6%) — / —
future-genetic 21 ✅ 2106 ms (88 iters) ✅ 2071 ms (90 iters) ≈ -1.7% (±2.7%) — / —
naive-bayes 21 ✅ 1257 ms (136 iters) ✅ 1340 ms (128 iters) ≈ +6.6% (±33.3%) — / —
reactors 21 ✅ 16574 ms (15 iters) ✅ 16423 ms (15 iters) ≈ -0.9% (±7.9%) — / —
reactors 25 ✅ 18460 ms (15 iters) ✅ 18165 ms (15 iters) ≈ -1.6% (±4.8%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2335 / 2304 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 5 / 6 8621 / 8635 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 1 8286 / 8482 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1285 / 1273 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1281 / 1282 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2946 / 3005 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 2 ✅ / 2874 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 4 3524 / 3523 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 7 / 3 3478 / 3496 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1718 / 1560 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1863 / 1781 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervm76f27
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 18.1.3 (1ubuntu1)
Date:Fri Aug 21 14:18:31 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs1
Logic error
Dereference of null pointer1

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Logic errorDereference of null pointerprofiler.hfindLibraryByAddress52228

Stale JMC IItemCollection usage predated the jafar migration on main
and caused a compile failure once this branch was rebased onto it.
@dd-octo-sts

dd-octo-sts Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 26ae49b9

@kaahos
kaahos marked this pull request as ready for review August 17, 2026 14:03
@kaahos
kaahos requested a review from a team as a code owner August 17, 2026 14:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a68d3f22c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

Sphinx Review found 2 critical/high severity finding(s) that must be addressed.

Comment thread ddprof-lib/src/main/cpp/javaApi.cpp

if (_thread_filter.enabled()) {
_thread_filter.unregisterThread(slot_id);
if (slot_id >= 0) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

[Sphinx Review — MEDIUM] onThreadEnd() now unconditionally calls _thread_filter.unregisterThread()/unregisterThreadByTid() (both mutex-guarded) for every JVMTI ThreadEnd event whenever any profiling engine is active, instead of being gated behind the thread-filter registry's active/enabled state as before.

Suggestion: Gate the else-branch (and ideally both branches) behind _thread_filter.registryActive() (or a cheaper lock-free check) so thread teardown for profiling sessions that never use the registry does not pay for a mutex acquisition.

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/profiler.cpp
Comment thread ddprof-lib/src/main/cpp/profiler.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.h
Comment thread ddprof-lib/src/main/cpp/wallClock.cpp
Comment thread ddprof-lib/src/main/cpp/wallClock.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java Outdated
@jbachorik jbachorik added the sphinx:critical Sphinx: critical — human review required label Aug 18, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from af36f26 to 2a142a6 Compare August 18, 2026 18:41
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 2a142a6 to 391fdd2 Compare August 18, 2026 18:58
@kaahos
kaahos requested a review from jbachorik August 18, 2026 19:03
kaahos and others added 5 commits August 19, 2026 10:05
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts:
#	ddprof-lib/src/main/cpp/javaApi.cpp
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 5c765fa to 1b94815 Compare August 20, 2026 13:01

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you, please, also take a look if it makes sense to add new adversaries in the chaos tests, or some fuzzing tests?

profiler.stop();
profilerStarted = false;
}
profiler.clearTraceContext();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QQ: Any reason this is done unconditionally? Eg. regardless of whether profilerStarted == true?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was originally done this way to prevent any garbage coming from another test interferes with this test. But you're right in a sense that no test currently writes context without calling start(). I'd still keep it like that though as it is the most defensive way of doing.

Comment thread ddprof-lib/src/main/cpp/engine.h Outdated
Comment thread ddprof-lib/src/main/cpp/frames.h
jint num_frames) {
// The source and destination commonly refer to the two views of the same
// CallTraceBuffer union. Read both source fields before either write.
for (jint i = 0; i < num_frames; ++i) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a premature optimization, but did you consider making the code friendlier to potential vectorization?
Probably not necessary to act upon this immediately, unless the benchmarks would show this function becoming a hotspot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I didn't for now, but it is definitely something to consider.

Comment thread ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java Outdated
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp
Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/wallClockCandidateSelector.h
@kaahos
kaahos requested a review from jbachorik August 21, 2026 14:03
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from e0ec41e to 26ae49b Compare August 21, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sphinx:critical Sphinx: critical — human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants