feat(power): multinode measured-power aggregation

[arygupt]

Summary

Extends single-node measured-power aggregation (#1558) to multinode srt-slurm benchmarks. Wires per-node perf_samples_<host>.csv from srt-slurm's PR #35 perfmon through the launcher into process_result.py → aggregate_power.py, which now namespaces local GPU indices per source CSV stem so each node's local indices 0..N-1 don't collapse across nodes.

Backward compatible: aggregate_power() accepts both Path and Iterable[Path]; single-CSV callers (single-node start_gpu_monitor path) are unchanged. csv_path param name preserved.

Pipeline

srt-slurm perfmon (PR #35 on NVIDIA/srt-slurm, layered on
  SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon)
  → perf_samples_<host>.csv in outputs/<job>/logs/ on shared NFS
  → launch_gb300-cw.sh exports GPU_METRICS_CSV_GLOB to $GITHUB_ENV
  → process_result.py expands glob → aggregate_power.run() with list
  → aggregate_power.py emits cluster-wide avg_power_w + joules_per_*_token
  → InferenceX-app ETL auto-captures (no schema change)

Files

• utils/aggregate_power.py — csv_path widened to Path | Iterable[Path]. Per-source GPU-id namespacing only kicks in for 2+ sources so single-node num_gpus is unchanged. CLI adds --csv-glob (mutually exclusive with --csv).
• utils/process_result.py — bridge GPU_METRICS_CSV_GLOB env var. Glob takes precedence over single GPU_METRICS_CSV when both are set.
• runners/launch_gb300-cw.sh — point dynamo-sglang at our srt-slurm fork, append monitoring: block to each recipe post-copy (idempotent), write GPU_METRICS_CSV_GLOB to $GITHUB_ENV after the job.
• utils/test_aggregate_power.py — 8 new multinode cases: per-source namespacing, sub-second clock drift, asymmetric prefill/decode power, missing-CSV silent skip, backward-compat single-path-in-list, Iterable acceptance, E2E with list.
• utils/test_process_result.py — 3 new cases: glob aggregation, precedence over single CSV, empty-match falls through.

Test plan

• 36/36 aggregator tests pass (28 existing + 8 new)
• 28/28 process_result tests pass (25 existing + 3 new)
• nvidia-smi inside sglang container on real gb300-cw emits expected columns (timestamp, gpu, power_w) — verified manually with srun --container-image=...sglang...sqsh nvidia-smi --query-gpu=...
• First E2E multinode sweep produces avg_power_w + joules_per_*_token in the agg JSON (pending perf-changelog.yaml entry + sweep-enabled label)
• num_gpus in agg JSON matches prefill_gpus + decode_gpus from launcher (validates per-source namespacing — without the fix, num_gpus would equal a single node's gpus_per_node)
• Chart at inferencex.semianalysis.com renders the new data via ?unofficialrun=<run_id>

Depends on

SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon (pinned by the launcher). Tracks NVIDIA/srt-slurm PR #35 head; will rebase to upstream main once #35 merges.

---

Note

Medium Risk
Touches benchmark launchers, result ETL, and multinode job lifecycle; failures are mostly best-effort for telemetry, but incorrect glob/recipe injection could silently omit power data or mis-attribute energy on disagg runs.

Overview
This PR extends measured GPU power and telemetry from single-node runs into multinode benchmarks on both NVIDIA (srt-slurm / gb300-cw) and AMD MI355X (amd_utils SLURM) paths, then patches aggregated results with cluster-wide and per-worker metrics.

NVIDIA: launch_gb300-cw.sh pins a srt-slurm fork with per-node perfmon, recursively injects monitoring: enabled into overlaid recipe YAMLs (fixing a flat-glob silent no-op), and sets GPU_METRICS_CSV_GLOB after the job. aggregate_power.py accepts multiple CSVs, namespaces GPU indices per source file, parses perf_samples_<role>_w<idx>_<host>.csv, and adds temp/util/memory, workers[], and disagg per-stage joules (joules_per_input_token, decode-attributed joules_per_output_token, prefill_avg_power_w / decode_avg_power_w). process_result.py prefers the glob over single-CSV when set and passes disagg through.

AMD: Each disagg node starts start_perf_monitor via benchmark_lib.sh (richer amd-smi sampling, shared perfmon filenames), wired through job.slurm and server_sglang.sh / server_vllm.sh; launch_mi355x-amds.sh collects CSVs early, exports the glob, and hardens squeue polling. CI pre/post-run sudo rm -rf benchmark_logs avoids root-owned checkout failures on cancelled runs.

Large test additions lock multinode, disagg, vendor CSV shapes, and process_result glob behavior.

^{Reviewed by Cursor Bugbot for commit 6849229. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26534043069
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26534043069

[claude]

🔴 When GPU_METRICS_CSV_GLOB is set but matches no files, _csv_arg stays None and the code falls through to the single-CSV candidate list (GPU_METRICS_CSV, gpu_metrics.csv, /workspace/gpu_metrics.csv) — contradicting the comment at lines 145-148 that the glob 'Takes precedence over the single-CSV fallback'. On a persistent self-hosted runner with a stale /workspace/gpu_metrics.csv from a prior single-node run (or a leaked GPU_METRICS_CSV env var), a multinode run whose perfmon failed on every node would silently patch wrong single-node avg_power_w / joules_per_*_token values into the multinode agg JSON. Fix: when _glob_pattern is truthy, skip the single-CSV fallback regardless of whether the glob matched anything.

Extended reasoning...

The contract violation

The block at utils/process_result.py:142-159 documents the precedence contract clearly:

Takes precedence over the single-CSV fallback — if the launcher set the glob, the run was multinode and there is no single-CSV fallback to make.

But the implementation only honors that contract when the glob actually matches files. On empty match:

_csv_arg = None
_glob_pattern = os.environ.get('GPU_METRICS_CSV_GLOB')
if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        print(..., file=sys.stderr)   # warns but doesn't prevent fallthrough

if _csv_arg is None:                  # still None — falls into single-CSV branch
    _csv_candidates = [
        os.environ.get('GPU_METRICS_CSV'),
        'gpu_metrics.csv',
        '/workspace/gpu_metrics.csv',
    ]
    ...

The else branch just logs; _csv_arg stays None, and the next if _csv_arg is None block consults the single-CSV candidates.

Step-by-step proof on a persistent self-hosted runner

1. Single-node run on gb300-cw_N completes successfully. benchmarks/benchmark_lib.sh exports GPU_METRICS_CSV=/workspace/gpu_metrics.csv (it lives in gpu_metrics.csv in cwd too). The file is left behind because the runner is persistent across jobs.
2. Next job is a multinode dynamo-sglang sweep. runners/launch_gb300-cw.sh (lines 297-318) writes GPU_METRICS_CSV_GLOB=$LOGS_DIR/perf_samples_*.csv to $GITHUB_ENV — but only when perf_csv_count > 0. Suppose perfmon failed to start on every node (srt-slurm PR [NVIDIA] Reduce B200 Runs & add B200 FP4 Docker Script #35 had startup issues, host driver mismatch, etc.) — perf_csv_count would be 0 and the glob env var would not be written. Fine — that path is safe.
3. However, suppose perfmon CSVs were written at the end of the job (so the launcher writes the GLOB), but a downstream cleanup hook between launcher and process_result.py removed them, OR srt-slurm wrote the CSVs to a different path on a subsequent retry, OR a persistent env var (GPU_METRICS_CSV_GLOB from a prior job) leaks in. The glob expansion in process_result.py returns empty.
4. process_result.py enters the else branch on line 155, prints a warning, and falls through. os.environ.get('GPU_METRICS_CSV') from the prior single-node job returns /workspace/gpu_metrics.csv (or gpu_metrics.csv in cwd is still there). Path(p).is_file() is True. _csv_arg = Path('/workspace/gpu_metrics.csv').
5. _aggregate_power_run is called with the stale single-node CSV.

Why the bench-window timestamp filter doesn't always save us

One verifier argued the start_unix <= ts <= end_unix filter at aggregate_power.py:177-178 would reject stale samples. That's true if the window comes from explicit Unix timestamps. But this PR adds two new fallback tiers in _load_bench_window:

• Tier 2: date field parsed as a UTC string (YYYYMMDD-HHMMSS).
• Tier 3: bench_result_path.stat().st_mtime — the bench JSON's own mtime, which is the current run's mtime, used as bench-end with start = end - duration.

The mtime tier is exactly the danger zone: on a persistent runner the bench JSON is freshly written, so its mtime is now. If the stale gpu_metrics.csv was also written recently (within the derived [mtime - duration, mtime] window — possible if the prior single-node run finished a few minutes ago), its samples do fall inside the window. Result: silent wrong avg_power_w and joules_per_*_token patched into the multinode agg JSON, which InferenceX-app's ETL auto-captures into the dashboard.

What the test misses

The accompanying test test_multinode_csv_glob_empty_match_falls_through_silently only asserts the no-stale-file case (asserts 'avg_power_w' not in patched). It does not stage a stale fallback CSV, so it can't catch the precedence violation. test_multinode_csv_glob_takes_precedence_over_single_csv only tests precedence when the glob matches.

Fix

One-line change in the empty-match branch:

if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        _csv_arg = []           # sentinel: glob attempted, fallback forbidden
        print(...)

if not _csv_arg:                # treats [] same as None for the downstream check, but…
    if _glob_pattern:
        pass                    # …skip single-CSV candidates when glob was attempted
    else:
        _csv_candidates = [...]
        _csv_arg = next(...)

Or more cleanly: guard the single-CSV block on not _glob_pattern instead of _csv_arg is None.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26547958720
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26547958720

[cursor]

Multinode num_gpus wrong without GPU column and clock drift

Low Severity

When multiple CSV paths are provided but none have a recognized GPU column (no match for _GPU_INDEX_COL_RE), num_gpus falls back to max(per_sample_row_count.values()). With multinode clock drift, each timestamp bucket only contains rows from a single node, so max returns one node's GPU count instead of the cluster total. This underestimates num_gpus and produces an incorrect total_system_energy_j, leading to wrong joules_per_*_token values in the agg JSON.

 

^{Reviewed by Cursor Bugbot for commit 8d30341. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26548110246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26548110246

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26593269421
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26593269421

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26600823211
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26600823211

[cursor]

Heterogeneous CSV schemas produce wrong cluster-wide metric averages

Medium Severity

When CSVs have heterogeneous schemas (e.g. one node's CSV has a temperature.gpu column but another's doesn't), _avg_per_gpu for optional metrics like temp/util/mem divides by the total GPU count in the bucket (per_sample_gpus) rather than by the number of GPUs that actually reported that metric. The per_sample_gpus set is populated from all sources regardless of which metrics they carry, so the denominator is inflated by GPUs that contributed no data for that metric, silently halving (or worse) the reported cluster-wide avg_temp_c, avg_util_pct, or avg_mem_used_mb.

Additional Locations (1)

• utils/aggregate_power.py#L350-L354

 

^{Reviewed by Cursor Bugbot for commit dea49cd. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26603522983
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26603522983

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26604906530
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26604906530

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26605086756
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26605086756

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26605402511
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26605402511

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 488ce46. Configure here.}

[cursor]

Double CSV I/O: every file read twice unnecessarily

Low Severity

The run() function calls aggregate_metrics(paths, start, end) and then aggregate_power_by_worker(paths, start, end) unconditionally. Both internally call _read_samples() on every CSV path, meaning every perfmon CSV is opened, parsed, and filtered twice. For multinode runs with many large per-node CSVs, this doubles the I/O and CPU cost of the aggregation step. The first call's parsed samples could be reused by the second.

Additional Locations (1)

• utils/aggregate_power.py#L709-L710

 

^{Reviewed by Cursor Bugbot for commit 488ce46. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26606436666
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26606436666