Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341) - #644
Draft
jbachorik wants to merge 23 commits into
Draft
Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341)#644jbachorik wants to merge 23 commits into
jbachorik wants to merge 23 commits into
Conversation
Contributor
|
🔗 Commit SHA: 9428745 | Docs | View more details | Give us feedback! |
Contributor
CI Test ResultsRun: #32148889778 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-08-18 14:59:38 UTC |
Contributor
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/127431974 ❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos |
Contributor
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 10, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 14, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 14, 2026 12:03
700d838 to
9972bff
Compare
Contributor
Benchmark Results (commit 9972bff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a636398)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f2d8978)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a48899b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10376 ms (21 iters) | ✅ 10228 ms (21 iters) | ≈ -1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8829 ms (24 iters) | ✅ 8859 ms (24 iters) | ≈ +0.3% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6001 ms (33 iters) | ✅ 5928 ms (33 iters) | ≈ -1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ +0.1% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2687 ms (70 iters) | ✅ 2761 ms (68 iters) | 🔴 +2.8% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2805 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2047 ms (91 iters) | ✅ 2116 ms (88 iters) | 🔴 +3.4% | — / — |
| future-genetic | 25 | ✅ 2000 ms (93 iters) | ✅ 2083 ms (89 iters) | 🔴 +4.2% | — / — |
| naive-bayes | 21 | ✅ 1256 ms (136 iters) | ✅ 1253 ms (136 iters) | ≈ -0.2% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1016 ms (169 iters) | ≈ 0% (±31.3%) | — / — |
| reactors | 21 | ✅ 16870 ms (15 iters) | ✅ 16463 ms (15 iters) | ≈ -2.4% (±7.3%) | — / — |
| reactors | 25 | ✅ 18270 ms (15 iters) | ✅ 18333 ms (15 iters) | ≈ +0.3% (±4.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1988 / 2024 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2263 / 2395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 8440 / 8679 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8283 / 8410 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1288 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 1250 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2968 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2843 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3474 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3495 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1726 / 1721 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1893 / 1750 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 787f29e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10215 ms (21 iters) | ✅ 10475 ms (21 iters) | ≈ +2.5% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8843 ms (24 iters) | ✅ 8861 ms (24 iters) | ≈ +0.2% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ -0.3% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5479 ms (36 iters) | ≈ -0.7% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2693 ms (70 iters) | ✅ 2836 ms (66 iters) | 🔴 +5.3% | — / — |
| fj-kmeans | 25 | ✅ 2820 ms (66 iters) | ✅ 2837 ms (66 iters) | ≈ +0.6% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2059 ms (90 iters) | ✅ 2070 ms (89 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2089 ms (89 iters) | ✅ 2130 ms (87 iters) | ≈ +2% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1313 ms (131 iters) | ≈ +2.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 988 ms (173 iters) | ≈ -2.1% (±31.4%) | — / — |
| reactors | 21 | ✅ 16540 ms (15 iters) | ✅ 16155 ms (15 iters) | ≈ -2.3% (±7.1%) | — / — |
| reactors | 25 | ✅ 18545 ms (15 iters) | ✅ 18816 ms (15 iters) | ≈ +1.5% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1987 / 1990 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2238 / 2323 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8290 / 8529 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8323 / 8386 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1299 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1274 / 1299 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2920 / 2982 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2886 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3505 / 3499 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1593 / 1669 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1849 / 1940 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit fede524)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit ef0f5c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 1299316)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10336 ms (21 iters) | ✅ 10245 ms (21 iters) | ≈ -0.9% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8868 ms (24 iters) | ✅ 8789 ms (24 iters) | ≈ -0.9% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5968 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ +0.1% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5510 ms (36 iters) | ✅ 5537 ms (36 iters) | ≈ +0.5% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2696 ms (69 iters) | ✅ 2822 ms (66 iters) | 🔴 +4.7% | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2746 ms (68 iters) | 🟢 -3.4% | — / — |
| future-genetic | 21 | ✅ 2068 ms (90 iters) | ✅ 2051 ms (90 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2134 ms (87 iters) | ✅ 2059 ms (90 iters) | 🟢 -3.5% | — / — |
| naive-bayes | 21 | ✅ 1318 ms (130 iters) | ✅ 1253 ms (136 iters) | ≈ -4.9% (±31.8%) | — / — |
| naive-bayes | 25 | ✅ 998 ms (171 iters) | ✅ 1025 ms (167 iters) | ≈ +2.7% (±32.1%) | — / — |
| reactors | 21 | ✅ 16264 ms (15 iters) | ✅ 16625 ms (15 iters) | ≈ +2.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 17926 ms (15 iters) | ✅ 18917 ms (15 iters) | ≈ +5.5% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1917 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2285 / 2434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8503 / 8628 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8331 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1287 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1277 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3037 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2897 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 12 | 3507 / 3493 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 5 | 3446 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1765 / 1665 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1855 / 1920 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 16, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 16, 2026 11:31
1299316 to
baba070
Compare
Contributor
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
3 tasks
Contributor
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 17, 2026 11:39
2033006 to
02a0a93
Compare
Contributor
Benchmark Results (commit 02a0a93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125255579 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit ed6525a · Any feedback? Reach out in #deveng-pr-agent |
Contributor
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 10, 2026 12:41
da35063 to
9b5fa79
Compare
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b5fa79851
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return; | ||
| } | ||
| _thread = thread; | ||
| _running.store(true, std::memory_order_release); |
There was a problem hiding this comment.
If pthread_create schedules the new thread before the parent reaches this store, threadLoop() attaches and immediately evaluates while (_running.load(...)) against the initial false, then detaches and exits. The parent subsequently publishes true, leaving the tracker reporting that it is running even though no BFS passes will execute for the entire recording; use a startup handshake or otherwise ensure the worker cannot test the flag before startup is published.
Useful? React with 👍 / 👎.
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 13, 2026 08:24
64a79a6 to
cba3ca6
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…chains Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a manual VMStructs-based heap walk that discovers reference chains from GC roots to live-heap sample objects, recorded as datadog.ReferenceChain / datadog.ReferenceChainAbandoned JFR events. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reset class-count counters in restartSearch(), fix null-base pointer UB in symbols_linux.cpp, gate JNI ref-minting in foldKlassCountsLocked() on allow_resolve, add totalHops to the ReferenceChain JFR event to avoid silent truncation, make LivenessTracker::_gc_generations atomic, widen per-klass population counters to u32, restore deleted FrameType tests, and correct stale root-discovery comments/docs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the CodeQL Autobuild failure: this test still used JMC's IItemCollection/IItem after AbstractProfilerTest's verifyEvents() switched to the jafar-backed JfrEvents API. Adds a JfrEvents overload of ReferenceChainAssertions.findMatchForClass alongside the existing IItemCollection one, which LeakingCacheScenario/ReferenceChainJfrParserTest still use unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
selectLeakCandidates()'s per-klass hysteresis gate can take longer to trust a candidate than an aggressive, heap-wide leak has before OOM. LivenessTracker::secondsToOOM() projects exhaustion from the heap-floor ring's growth rate; hasLeakSignal() now bypasses the per-klass gate once that projection drops under OOM_URGENT_THRESHOLD_S (5min). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also bump leak-seeding loops from 10 to 15 epochs: LivenessTracker's hysteresis only starts counting consecutive_positive once ring_fill reaches 10, so exactly 10 seed calls could never clear the >=5 threshold. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ples A real GC firing between resetKlassPopulationForTest0() and shouldRunPassForTest0() could interleave a heap-floor sample with a real OS::nanotime() timestamp and real heap usage, corrupting secondsToOOM()'s projection. On musl/Liberica the GC timing made this race reproducible (8/8 musl jobs failed; glibc passed by luck). Add a debug-only test seam (setHeapFloorRecordingForTest0) that temporarily disables onGC()'s own recordHeapFloorSample() call so the test can seed the ring exclusively via heapFloorRecordForTest0().
The debug-only test seams had non-atomic writes that arm64's weak memory model may never publish to the BFS/GC thread: - _heap_floor_recording_disabled_for_test was a plain bool; the GC thread's read may never see the test thread's store, so a real GC sample with a real OS::nanotime() timestamp could still corrupt the ring. Now std::atomic with acquire/release. - klassPopulationResetForTest() used plain store() (relaxed) for the ring indices; the GC thread may never see the reset. Now storeRelease(). - _max_heap_bytes is a plain jlong written by setMaxHeapBytesForTest() from the test thread and read by secondsToOOM() on the BFS thread. Added an atomic mirror (_max_heap_bytes_for_test) with acquire/release for the debug test path; production path unchanged (written once before the BFS thread starts). Also moved the disabled-flag check from onGC() into recordHeapFloorSample() at the actual write point, so a GC already past onGC()'s gate still cannot corrupt the ring. Added recordHeapFloorSampleUnchecked() so the test seam can bypass the flag.
Older dd-java-agent builds compiled DatadogProfiler against the pre-1.45 JavaProfiler.setContextValue(int, CharSequence) signature. Commit 22c3a7e changed it to setContextValue(int, String) — source compatible but binary incompatible, causing NoSuchMethodError at runtime when a patched agent jar pairs the new JavaProfiler with an old DatadogProfiler. The overload delegates to setContextValue(int, String) after toString() — the value is immediately converted to UTF-8 by the cache either way, so no extra allocation or correctness cost.
The overload was a temporary shim for older dd-java-agent builds compiled against the pre-1.45 CharSequence signature. A dd-java-agent built against current java-profiler main uses setContextValue(int, String) directly, so the overload is unnecessary.
The AggressiveLeakReferenceChainTest failed on musl CI because resetSearchStateForTest() spent the previous test's _search_pain_ms into _pain_budget but did not reset the budget itself. On fast CI runners (musl, small heap, no GC pauses) the wall-clock drain between tests was insufficient, leaving the budget non-zero and blocking canAffordNewSearch() -> shouldRunPass() returned false before ever reaching hasLeakSignal(). Cache the refill rate from start() in _pain_budget_refill_rate and rebuild the budget with it in resetSearchStateForTest() so every test starts from zero debt.
The fixed defaults (budget=1000, ttl=60s, framecap=65536) are tuned for a small heap. On a 4 GiB heap the BFS explores only ~60k edges before the 60s TTL expires, abandoning every search without meaningful progress. Add ReferenceChainTracker::autoTuneDefaults() which scales unset defaults from the process's max heap size and available processor count, called from start() after LivenessTracker resolves max heap but before the frontier table is constructed. Only overrides defaults the operator did not set explicitly (tracked by _reference_chains_tuned_mask in Arguments). Tuning logic: budget: sqrt(heap_mib / 512) * default (4GiB -> 2x, 16GiB -> 4x) firstpass: 10x budget (one-shot root enumeration can afford more) ttl: (heap_mib / 512) * default, clamped [60s, 30min] framecap: budget-scaled (larger budget -> larger frontier) pausetarget: 1 + (nprocs-1)/3 * default, capped 50ms painbudget: 1 + (nprocs-1)/4 * default, capped 5% hop_cap is left alone (bounds chain depth, not breadth).
…scalation The TTL was a time-based bound, but the search's completion is a graph-size problem. On a 4 GiB heap, the BFS needs ~2000 passes at 2000 edges/pass — the TTL (auto-tuned to 840s) kills the search at pass 336, frontier only half full, making progress but not done. Then the pain budget drains, restarts, and repeats — burning CPU forever without producing chains. Proposal 1 — progress-based termination: Replace the TTL abandonment with a no-progress detector. Abandon only after NO_PROGRESS_PASS_LIMIT (30) consecutive passes with zero new frontier entries admitted (genuinely stuck, not just slow). A large heap takes more passes simply because there are more objects — that is not 'stuck'. Suppressed when isUrgent() (secondsToOOM() < OOM_URGENT_THRESHOLD_S): the search must complete before the app OOMs, bounded only by the per-pass STW pause SLO. Proposal 2 — urgency-driven budget escalation: When isUrgent(), raise the per-pass pause target to URGENT_PAUSE_TARGET_MS (50ms) and tighten cadence to URGENT_CADENCE_NS (10ms) so passes run back-to-back. Also raise _budget 4x so the PID controller can let each pass explore more edges. The total STW time is the same, but the search converges in fewer, longer passes. The _pass_deadline_ns and updatePacing() now use _effective_pause_target_ms instead of the static _pause_target_ms. resetSearchStateForTest() resets _passes_since_last_progress. Updated the TTL unit test to verify the new wiring.
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 18, 2026 14:05
b56c65c to
979bd01
Compare
Proposes replacing the whole-graph forward BFS with a pruned forward BFS that treats leaked objects as leaves and terminates early when all leaked candidates are found. Key design decisions (from review): - Marker tag must be negative (class-tag range) to avoid collision with frontier tags (positive jlong from _next_tag). - Do not return JVMTI_VISIT_ABORT when pruning -- that aborts the entire FollowReferences walk, not just the candidate subtree. Simply skip the admitObject() call instead. - Pre-tag candidates in threadLoop()'s first iteration, not in start() -- keeps everything on the BFS thread. - Cleanup via GetObjectsWithTags(MARKER_TAG) -- don't re-resolve jweak references (candidate may have died). - resetSearchStateForTest() must reset the candidate set. - Cost claim softened to 10-100x (worst case is same as whole-graph BFS if candidates are deep).
This reverts commit bd81ff5.
Replace the whole-graph forward BFS with a pruned forward BFS that treats leaked objects as leaves and terminates early when all leaked candidates are found. Phase 1: Candidate state fields on ReferenceChainTracker - MARKER_TAG_BASE (negative, disjoint from frontier tags) - _candidate_count, _candidate_found_bits, _candidate_tags[], _candidate_frontier_tags[] - Init in start(), reset in resetSearchStateForTest() Phase 2: Pre-tag candidates in threadLoop() - Before first runPass(), call selectLeakCandidates() and SetTag(obj, MARKER_TAG_BASE - i) for each candidate. - Gated on gcGenerationsEnabled() (no-op without it). Phase 3: Prune heapReferenceCallback() at candidates - Check *tag_ptr <= MARKER_TAG_BASE (marker tag, negative). - Record chain link via FrontierTable::insert() with the marker tag as the frontier key (JVMTI SetTag overwrites, so the marker tag doubles as the buildChainEvent() lookup key). - Skip admitObject() (don't enqueue children). - Do NOT return JVMTI_VISIT_ABORT (aborts entire walk). Phase 4: Early termination - After each pass, if popcount(_candidate_found_bits) == _candidate_count, transition to COMPLETED. Phase 5: Cleanup - On completion/abandonment, GetObjectsWithTags for each marker tag, SetTag(obj, 0), DeleteLocalRef, Deallocate. - Reset candidate state. Phase 7: pollWatchedTargets() canary path - When getTag() returns a marker tag (negative), use it directly for buildChainEvent() lookup instead of the normal frontier tag path.
With canary pruning, total STW is <20ms per 60s recording in practice. Use that wiggle room: - Remove cadence sleep between passes when canary search is active (candidates still to find). shouldRunPass() returns true immediately for an active canary search, bypassing the cadence gate. The PID controller self-regulates per-pass STW pause. - Frontier cap hit no longer abandons the search immediately. The search stays RUNNING: existing frontier entries may still lead to candidates, and the no-progress detector handles abandonment. The frontier cap is now a memory bound, not a correctness bound. Updated tests: - FrontierCapAbandonsSearchAndReleasesTags -> FrontierCapHitDoesNotAbandonImmediately (verifies search stays RUNNING after cap hit). - ReleaseSearchTagsFailureBlocksTagReuseUntilItSucceeds updated to use no-progress abandonment instead of frontier cap (which no longer abandons). - ReferenceChainsTestAccessor::reset() now resets _passes_since_last_progress and candidate fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341). A
ReferenceChainTrackerruns adedicated BFS thread that tags reachable objects with JVMTI object tags,
walks the heap incrementally across GC epochs, and records the referrer-type
chain back toward a GC root (labelled with its GC-root kind). It is bridged
to
LivenessTracker::selectLeakCandidates()'s population-slope leakdetection: when a still-live sampled object starts to look leak-shaped, its
chain is reconstructed from the frontier table and emitted to JFR.
flowchart TD LT["LivenessTracker::selectLeakCandidates (population-slope ranking)"] -->|"ranked klass candidates"| PWT["ReferenceChainTracker::pollWatchedTargets"] BFS["BFS thread: threadLoop"] -->|"shouldRunPass gate: GC epoch advanced or cadence elapsed"| RP["runPass"] RP -->|"every pass"| MW["runPassManualWalk: IterateOverReachableObjects seeds roots"] MW --> EF["expandFrontier: batched array-holder FollowReferences"] EF --> FT["FrontierTable: FRONTIER / EXPANDED / EDGE / ABANDONED"] FT -->|"tag already set on a leak-candidate instance"| PWT PWT -->|"buildChainEvent"| RC["cacheResolvedChain: one entry per klass id, cap 256"] RC -->|"Profiler::dump, snapshot without clearing"| DR["drainPendingChainEvents"] DR --> JFR["datadog.ReferenceChain / datadog.ReferenceChainAbandoned"]Heap-walk mechanism (the main design decision in this branch): every pass
takes a bounded manual walk (
runPassManualWalk()) — pure JVMTI(
IterateOverReachableObjectsfor root enumeration +expandFrontier()forincremental frontier expansion), resumable across passes, and bounded so its
safepoint pauses stay small on every collector, including ZGC. The walk
issues only JVMTI heap calls, which run inside the
VM_HeapWalkOperationsafepoint and honor ZGC's load barriers, so concurrent relocation cannot
corrupt it — it reads no raw oop.
expandFrontier()'s batching, concretely: instead of oneFollowReferencescall per frontier entry, up to
budgetpending tags are resolved in oneGetObjectsWithTagscall, packed into a single JNI array (holder), andexpanded via exactly one
FollowReferences(initial_object = holder)call —so one BFS level is discovered per VM-safepoint operation rather than one
safepoint per entry:
sequenceDiagram participant BFS as BFS thread participant JVMTI as JVMTI participant JNI as JNI holder array participant CB as heapReferenceCallback BFS->>BFS: pull up to budget tags from front of _pending_expand BFS->>JVMTI: GetObjectsWithTags, resolve which tags are still live JVMTI-->>BFS: live jobject references BFS->>JNI: EnsureLocalCapacity, then NewObjectArray to build holder alt exception, EnsureLocalCapacity failure, or null holder BFS->>BFS: ctx.truncated = true, retry this batch next pass else holder built successfully BFS->>JNI: SetObjectArrayElement per resolved object BFS->>JVMTI: FollowReferences, initial_object = holder JVMTI->>CB: heapReferenceCallback per outgoing edge CB-->>JVMTI: descend only for batch_tags boundary objects JVMTI-->>BFS: one BFS hop expanded for the whole batch BFS->>BFS: markExpanded, admitObject appends children to _pending_expand endexpandFrontier()'s JNI/JVMTI error handling is defensive by construction: anull
holder, a pending JNI exception afterNewObjectArray/SetObjectArrayElement, or anEnsureLocalCapacityfailure all setctx.truncated = true(retry next pass) instead of marking the batchpermanently
EXPANDED; a failedjava/lang/Objectclass resolution withpending work also forces
truncated = true, sorunPass()cannot mistakeit for
SearchState::COMPLETED.Termination and pacing — an unbounded traversal could otherwise stall a
GC safepoint or run forever:
stateDiagram-v2 direction LR [*] --> RUNNING RUNNING --> COMPLETED: frontier drained,<br/>no truncation this pass RUNNING --> ABANDONED: frontier-size cap hit,<br/>or wall-clock TTL exceeded<br/>with work still pending COMPLETED --> RUNNING: restartSearch ABANDONED --> RUNNING: restartSearch note right of ABANDONED SearchAbandonReason (FRONTIER_CAP/TTL) records which cutoff fired, for the T_REFERENCE_CHAIN_ABANDONED JFR event end note note right of RUNNING restartSearch only fires once a leak candidate is seen and PainBudget allows it end notetreated as immediate search abandonment, not per-pass truncation) —
configured via
referencechains=true:hops=N:budget=N:ttl=N:framecap=N(plus
firstpassbudget=Nfor the first pass's own budget). Negativehops/budget/framecapvalues are floored (an unfloored negativehopswould otherwise wrap to ~4e9 as au32, silently disabling the capit's meant to enforce); all three are also ceiling-clamped.
updatePacing(),pausetarget=Nms)adapts the effective per-pass budget and cadence toward a configured
safepoint-pause SLO.
restarted walk is very likely to still find the object) plus a
PainBudgetcooldown (painbudget=N, 0-100) — a leaky bucket overcumulative safepoint time spent, so a cheap search can restart sooner than
an expensive one. This closes a structural gap where a one-shot walk could
finish before population-trend detection had accumulated enough GC epochs
to flag a candidate, leaving anything allocated afterward permanently
undiscoverable.
LivenessTracker::selectLeakCandidates()) requiresa klass's population growth to clear both a magnitude bar and a floor-rise
bar for
LEAK_TREND_HYSTERESIS_BASE(5) consecutive qualifying epochs —lowered to
LEAK_TREND_HYSTERESIS_CORROBORATED(3) when the aggregatepost-GC heap floor is also rising — rather than trusting a single epoch's
positive slope. This closes a false-positive gap where an oscillating
("see-saw") population could otherwise trigger a search with no real
longer-term growth.
JFR persistence across dumps: a resolved chain is cached per klass id
(
_resolved_chains, capped at 256 entries, drop-not-evict once full — seeREFERENCE_CHAIN_EVENTS_DROPPED) and re-stamped into every subsequent dumpthe sample survives into (
drainPendingChainEvents()snapshots withoutclearing, mirroring how
LivenessTrackerre-emits live-object samples), so along-lived leak's chain is present in each JFR chunk rather than only the
chunk active when it was first reconstructed. Verified end-to-end in
ExternalProcessReferenceChainTest.Motivation:
PROF-15341 — give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive",
without regressing safepoint-pause behaviour on low-pause collectors.
Additional Notes:
See
doc/reference-chains-design.mdanddoc/reference-chains-collection-summary.mdfor the full design and leak-detection mechanism. Per-pass
budget/hop-cap/TTL/frontier-cap/pause-target/pain-budget defaults are round,
unbenchmarked placeholders pending a measurement pass — no JMH harness for
this feature exists yet, though
utils/has repro, parameter-sweep, andJFR-report shell/Python tooling used to characterise pause behaviour ad hoc.
Earlier planning/proposal docs (implementation plan, remaining-work plan,
benchmark plan, and an alternative VMStructs-walk design) described approaches
or work that were superseded or shipped differently than planned, so they were
dropped from this PR (kept locally, not committed) rather than left to drift
further from the code.
How to test the change?:
./gradlew :ddprof-lib:gtestDebug— native unit suite, includingReferenceChainsTest/ReferenceChainsBfsTest/ReferenceChainsTagTest/FrontierTableTest/PollWatchedTargetsTest/ResolvedChainCacheTest,PainBudgetTest,SearchRestartTest(restart gate),ReferenceChainJfrRoundtripTest(JFR encode/decode),ArgumentsTest(
referencechains=sub-option parsing, includinghops/budget/framecapclamping behaviour), and the
LivenessTracker/SelectLeakCandidatesbridging tests.
./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"—genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM, asserts the reconstructed chain's leaf class, and
confirms the chain re-emits into three fresh JFR dumps (across-dumps
persistence).
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"—in-process coverage for the walk engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code