Skip to content

Remove USS sampling from the benchmark memory recorder - #7837

Draft
fatimaanes wants to merge 2 commits into
isaac-sim:developfrom
fatimaanes:fix/benchmark-drop-uss-sampling
Draft

fatimaanes wants to merge 2 commits into
isaac-sim:developfrom
fatimaanes:fix/benchmark-drop-uss-sampling

Conversation

@fatimaanes

@fatimaanes fatimaanes commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Description

MemoryInfoRecorder sampled Unique Set Size on every update via
psutil.Process.memory_full_info(). That call walks the process page tables, so its cost
scales with resident size rather than being a constant counter read.

BenchmarkMonitor drives this recorder once per second from a background thread, and that
thread is started inside the timed region in all nine benchmark entry points
(entrypoints/runtime.py:186 plus the eight rl_games / rsl_rl / sb3 / skrl train and play
entry points). USS collection therefore ran concurrently with the workload each benchmark
was measuring.

Measured on this host (AMD EPYC 9124, Linux 6.8.0-87, psutil 5.9.8 per uv.lock), one
MemoryInfoRecorder.update() at 8 GB resident:

before after
update() median 71.42 ms 0.012 ms
allocating worker throughput (6 s) 12,320 ops 12,999 ops (+5.5%)
measurements emitted 12 8

RSS and VMS are read from memory_info(), which is a cheap counter read and is unaffected.

Why remove rather than budget or sample less often

An earlier proposal timed each sample against a budget and latched USS collection off once
it exceeded it. That keeps the metric nominally present while making it unreliable:

  • The over-budget sample is still folded into the Welford statistics before the latch
    fires, so the reported mean/peak becomes an early-run prefix of a growing process rather
    than a run statistic.
  • The budget is wall-clock around a call that releases the GIL, so on Linux it measures GIL
    re-acquisition latency as much as query cost and can latch off on a small process under
    thread contention.
  • Whether the metric survives becomes a function of host memory size and page composition,
    so runs stop being comparable across machines.

Reducing the sampling interval has the same problem in weaker form: it lowers the duty cycle
but keeps an unbounded-cost call inside the timed region.

Because USS has no consumer (below), removing it avoids all of this and leaves no stale or
partial values behind.

The change is also deliberately confined to the recorder. The nine
with ... BenchmarkMonitor(benchmark, interval=1.0): lines are left byte-identical, because
downstream benchmark tooling pattern-matches the literal shape of those lines to inject
profiler capture anchors into Isaac Lab's entry points. Remedies that restructure the
with statement, rename the benchmark argument, or move sampling out of the monitor would
silently break that tooling; fixing the cost at its source does not.

Output change

These four measurements are no longer emitted:

  • System Memory USS
  • System Memory USS std
  • System Memory USS peak
  • System Memory USS n

MemoryInfoRecorder now always emits exactly 8 measurements (RSS and VMS, each with mean,
std, peak and n), so the emitted set is no longer platform-dependent.

I searched the repository, its history, docs, test fixtures and serialized output
expectations for consumers before removing anything:

  • capture.py:346-348 reads only System Memory RSS, RSS std and RSS peak.
  • The typed schema (benchmark/schema.py) has no USS field.
  • formatters.py passes measurements through generically, so a removed row cannot raise.
  • The console summary only matches names beginning Min/Max/Mean/Std.
  • No JSON fixture, golden file, or documentation page lists a USS field.

The only in-repo references were the producer itself and assertions/comments in
test_recorders.py, both updated here. System Memory USS peak appears in a historical
CHANGELOG.rst entry, which is left untouched as a record of the past release.

On the downstream side: the benchmark harness that runs these workloads collects its own USS
figure directly from psutil rather than reading Isaac Lab's measurement, so removing this
field does not deprive it of the metric. Previously recorded System Memory USS series remain
in historical dashboard data; because they were produced while the sampler was perturbing the
run, they are not comparable with post-change runs and should be treated as a separate
baseline rather than a continuous series.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update: no — no docs page documented the USS fields

Tests

Run with the repo .venv (Python 3.12.13, torch 2.11.0+cu128, psutil 5.9.8), pytest 9.1.1:

python -m pytest source/isaaclab/test/benchmark/test_recorders.py \
                 source/isaaclab/test/benchmark/test_memory_recorder_no_uss.py -q
# 73 passed in 4.64s

python -m pytest source/isaaclab/test/benchmark/ -q
# 11 failed, 346 passed, 1 skipped   (this branch)
# 11 failed, 327 passed, 1 skipped   (origin/develop e86463df8a, unmodified)

The failing set is byte-identical on both, so this branch introduces no regression. The 11
pre-existing failures are in test_api.py (10) and
test_asset_suite_runtime_semantics.py (1) and are unrelated to this change.

New regression coverage:

  • test_memory_full_info_is_never_called — monkeypatches psutil.Process.memory_full_info
    to raise, so reintroducing the call fails the suite.
  • test_no_uss_keys_in_runtime_data, test_get_data_measurement_names — no USS key or row
    is emitted.
  • test_rss_and_vms_means_are_tracked — scripted RSS/VMS values, asserting mean, peak and n.
  • test_benchmark_monitor_never_queries_uss, test_monitor_reports_no_recorder_exception
    a live BenchmarkMonitor over a real benchmark, asserting the recorder still updates, the
    monitor records no exception, and its thread is joined.
  • test_finalized_output_contains_no_uss, test_supported_formatters_still_serialize
    omniperf, json, osmo and summary all still write output containing no USS field.
  • test_entrypoints_keep_monitor_and_recorder_configuration — guards that each of the nine
    entry points still constructs a BenchmarkMonitor with use_recorders=True.

Lint and hooks:

ruff 0.14.10 check   -> All checks passed  (benchmark source + tests)
ruff 0.14.10 format  -> clean
codespell 2.4.1      -> clean
python tools/changelog/cli.py check --include-worktree
                     -> All modified packages have valid changelog fragments.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/isaaclab/changelog.d/
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

🤖 Generated with Claude Code

MemoryInfoRecorder called psutil.Process.memory_full_info() on every update.
That call walks the process page tables, so its cost scales with resident size:
measured at 71.4 ms per update on an 8 GB process (psutil 5.9.8, Linux 6.8).

BenchmarkMonitor drives this recorder once per second from a background thread
that is started inside the timed region of all nine benchmark entry points, so
the sample perturbed the workload the run was measuring.

USS had no consumer. capture.py maps only RSS into the typed bundle, the typed
schema has no USS field, the console summary cannot print it, and no in-repo
formatter or fixture reads it. RSS and VMS come from memory_info(), a cheap
counter read, and are unchanged.

Removes the USS state, sampling and the four "System Memory USS*" measurements
rather than retaining stale or partial values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Sep 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant