Remove USS sampling from the benchmark memory recorder - #7837
Draft
fatimaanes wants to merge 2 commits into
Draft
fatimaanes wants to merge 2 commits into
fatimaanes wants to merge 2 commits into
Conversation
MemoryInfoRecorder called psutil.Process.memory_full_info() on every update. That call walks the process page tables, so its cost scales with resident size: measured at 71.4 ms per update on an 8 GB process (psutil 5.9.8, Linux 6.8). BenchmarkMonitor drives this recorder once per second from a background thread that is started inside the timed region of all nine benchmark entry points, so the sample perturbed the workload the run was measuring. USS had no consumer. capture.py maps only RSS into the typed bundle, the typed schema has no USS field, the console summary cannot print it, and no in-repo formatter or fixture reads it. RSS and VMS come from memory_info(), a cheap counter read, and are unchanged. Removes the USS state, sampling and the four "System Memory USS*" measurements rather than retaining stale or partial values. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
MemoryInfoRecordersampled Unique Set Size on every update viapsutil.Process.memory_full_info(). That call walks the process page tables, so its costscales with resident size rather than being a constant counter read.
BenchmarkMonitordrives this recorder once per second from a background thread, and thatthread is started inside the timed region in all nine benchmark entry points
(
entrypoints/runtime.py:186plus the eight rl_games / rsl_rl / sb3 / skrl train and playentry points). USS collection therefore ran concurrently with the workload each benchmark
was measuring.
Measured on this host (AMD EPYC 9124, Linux 6.8.0-87, psutil 5.9.8 per
uv.lock), oneMemoryInfoRecorder.update()at 8 GB resident:update()medianRSS and VMS are read from
memory_info(), which is a cheap counter read and is unaffected.Why remove rather than budget or sample less often
An earlier proposal timed each sample against a budget and latched USS collection off once
it exceeded it. That keeps the metric nominally present while making it unreliable:
fires, so the reported mean/peak becomes an early-run prefix of a growing process rather
than a run statistic.
re-acquisition latency as much as query cost and can latch off on a small process under
thread contention.
so runs stop being comparable across machines.
Reducing the sampling interval has the same problem in weaker form: it lowers the duty cycle
but keeps an unbounded-cost call inside the timed region.
Because USS has no consumer (below), removing it avoids all of this and leaves no stale or
partial values behind.
The change is also deliberately confined to the recorder. The nine
with ... BenchmarkMonitor(benchmark, interval=1.0):lines are left byte-identical, becausedownstream benchmark tooling pattern-matches the literal shape of those lines to inject
profiler capture anchors into Isaac Lab's entry points. Remedies that restructure the
withstatement, rename thebenchmarkargument, or move sampling out of the monitor wouldsilently break that tooling; fixing the cost at its source does not.
Output change
These four measurements are no longer emitted:
System Memory USSSystem Memory USS stdSystem Memory USS peakSystem Memory USS nMemoryInfoRecordernow always emits exactly 8 measurements (RSS and VMS, each with mean,std, peak and n), so the emitted set is no longer platform-dependent.
I searched the repository, its history, docs, test fixtures and serialized output
expectations for consumers before removing anything:
capture.py:346-348reads onlySystem Memory RSS,RSS stdandRSS peak.benchmark/schema.py) has no USS field.formatters.pypasses measurements through generically, so a removed row cannot raise.Min/Max/Mean/Std.The only in-repo references were the producer itself and assertions/comments in
test_recorders.py, both updated here.System Memory USS peakappears in a historicalCHANGELOG.rstentry, which is left untouched as a record of the past release.On the downstream side: the benchmark harness that runs these workloads collects its own USS
figure directly from
psutilrather than reading Isaac Lab's measurement, so removing thisfield does not deprive it of the metric. Previously recorded
System Memory USSseries remainin historical dashboard data; because they were produced while the sampler was perturbing the
run, they are not comparable with post-change runs and should be treated as a separate
baseline rather than a continuous series.
Type of change
Tests
Run with the repo
.venv(Python 3.12.13, torch 2.11.0+cu128, psutil 5.9.8), pytest 9.1.1:The failing set is byte-identical on both, so this branch introduces no regression. The 11
pre-existing failures are in
test_api.py(10) andtest_asset_suite_runtime_semantics.py(1) and are unrelated to this change.New regression coverage:
test_memory_full_info_is_never_called— monkeypatchespsutil.Process.memory_full_infoto raise, so reintroducing the call fails the suite.
test_no_uss_keys_in_runtime_data,test_get_data_measurement_names— no USS key or rowis emitted.
test_rss_and_vms_means_are_tracked— scripted RSS/VMS values, asserting mean, peak and n.test_benchmark_monitor_never_queries_uss,test_monitor_reports_no_recorder_exception—a live
BenchmarkMonitorover a real benchmark, asserting the recorder still updates, themonitor records no exception, and its thread is joined.
test_finalized_output_contains_no_uss,test_supported_formatters_still_serialize—omniperf,json,osmoandsummaryall still write output containing no USS field.test_entrypoints_keep_monitor_and_recorder_configuration— guards that each of the nineentry points still constructs a
BenchmarkMonitorwithuse_recorders=True.Lint and hooks:
Checklist
pre-commitchecks with./isaaclab.sh --formatsource/isaaclab/changelog.d/CONTRIBUTORS.mdor my name already exists there🤖 Generated with Claude Code