From d222beea783aea3698aa1f95194fadabb408db29 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Thu, 17 Sep 2026 23:01:50 +0000 Subject: [PATCH] docs(eval): align GDPVal/TB2.1/SWE-bench/MRCR skills with upstream configs Aligns the evaluation skill's four benchmark recipes, config templates and the GDPVal SIF builder with nvidia-eval-factory-benchmarking `main` (92b5474). GDPVal is now GDPVal-AA v2 upstream, which is a different measurement: - metric key is the final-stage-scoped `comparison/stage_1/normalized_elo`; the flat key is emitted only on an exactly-complete final stage - 12-model reference ladder (human_gold and gptoss_20b deliberately disabled), two-stage fit (45 -> 220 tasks) with partial_completion gates - judge is a sampled 3-member panel, not a single model - Gym pin df3e201d, SIF python-3.13.gdpval.gym-80e4fc.sif, Python >= 3.13.14 Fixes `gdpval-sif.sh` for that pin: the v2 `gdpval.def` declares a `%files` section whose sources apptainer resolves against the build CWD, so the def can no longer be built on its own. The script now stages the def plus every source named in its `%files` block and builds from inside that directory. MRCR: canonical `parallelism` is 512, deliberately above server capacity, which invalidates the old `ceil(parallelism / num_instances / DP)` rule for `--max-num-seqs`; `limit_samples` now reaches the gym via a gated `++limit`. Terminal-Bench 2.1 / SWE-bench Verified: `http_pairs_dump` is first in the interceptor chain for both; sandbox region comes from `HARBOR_ECS_REGION`, whose default and ECR coupling differ between the two. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Chenjie Luo --- plugins/modelopt/skills/evaluation/SKILL.md | 20 ++- .../skills/evaluation/recipes/env.example | 20 ++- .../recipes/examples/example_eval_next.yaml | 42 +++-- .../recipes/examples/gym/example_gdpval.yaml | 57 ++++-- .../recipes/examples/gym/example_mrcr.yaml | 21 ++- .../tasks/aa_next/swebench_verified.md | 50 ++++-- .../tasks/aa_next/terminal_bench_2_1.md | 27 ++- .../evaluation/recipes/tasks/gym/gdpval.md | 73 +++++--- .../evaluation/recipes/tasks/gym/mrcr.md | 65 +++++-- .../evaluation/references/gym-gdpval.md | 168 +++++++++++++++--- .../skills/evaluation/references/nel-next.md | 33 +++- .../skills/evaluation/scripts/gdpval-sif.sh | 72 ++++++-- 12 files changed, 502 insertions(+), 146 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index da57288528f..4b574bfc53a 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -42,7 +42,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead: 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). -3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block. +3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block. 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r -c ` after (config-driven; see `references/nel-next.md`). Steps 1–9 below are currently validated with 0.2.6 — use them for everything else. @@ -65,18 +65,22 @@ tasks). If the user asks for GDPVal: (NVIDIA-internal: `modelopttools:eval-config` Step 3c); otherwise set `GDPVAL_SIF_DIR` in `.env` and build with `"$SKILL_DIR/scripts/gdpval-sif.sh"` (build-if-absent, no cross-cluster copy). Either way the mounted dir must contain - the file `GDPVAL_CONTAINER_PATH` names (template: `python-3.13.gdpval.sif`) — a + the file `GDPVAL_CONTAINER_PATH` names (canonical: + `python-3.13.gdpval.gym-80e4fc.sif`, the GDPval-AA v2 sandbox) — a name mismatch passes NEL's `test -d` check and the agent then silently runs - unsandboxed. Verify with `gdpval-sif.sh --check`. `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`, + unsandboxed. Verify with `gdpval-sif.sh --check`. The SIF, the Gym pin and the eval + image's Python move as **one unit** (reference: "Three things that move together"). + `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`, `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR`, and `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the config has a `pre_cmd`). Thinking mode is mandatory (non-thinking loses ~86%). 4. Run both dry-run and launch through `"$SKILL_DIR/scripts/nel-gdpval.sh"`; it enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale and avoids an unset `NEL_INVOCATION_ID` failure before client startup. - **`limit_samples` is inert on the gym path** (the gym runs all 220 tasks - regardless), so there is no cheap canary: watch the real run's first - ~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the - recipe's Canary section. + **`limit_samples` is inert on the GDPVal gym path** (the gym runs all 220 tasks + regardless; the config says so via `allow_limit_samples: false`), so there is no + cheap canary: watch the real run's first ~20–30 min for the SIF-sandbox line and + judge auth, and cancel if wrong. See the recipe's Canary section. (MRCR is the + exception — there `limit_samples` *does* reach the gym.) --- @@ -92,6 +96,8 @@ for an "AA" request. If the user asks for MRCR: 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it sets the context cap, dataset *and* metric prefix; the three are not comparable; set it in **both** `data_prep_params` and `collect_rollout_params`. + Upstream now carries 1M and 128K as two separate benchmarks with their own + manifests and metric keys — see the recipe. 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` + `transformers`; prepare fails without it) and diff --git a/plugins/modelopt/skills/evaluation/recipes/env.example b/plugins/modelopt/skills/evaluation/recipes/env.example index 2eb91e377f6..9a03c28fb58 100644 --- a/plugins/modelopt/skills/evaluation/recipes/env.example +++ b/plugins/modelopt/skills/evaluation/recipes/env.example @@ -40,6 +40,14 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # GDPVal (nemo_gym) also reuses INFERENCE_JUDGE_URL for its pairwise judge. # INFERENCE_JUDGE_URL=https:///v1 +# GDPVal COMPARISON mode only — upstream splits judge auth from INFERENCE_API_KEY and +# reaches two auxiliary endpoints. The rubric template in this skill reuses +# INFERENCE_API_KEY for the judge and needs none of these; the +# `modelopttools:eval-config` Step 3c conversion adds them. +# JUDGE_API_KEY= # 3-member judge panel auth +# INTEGRATE_NVAPI_KEY= # Qwen3-235B helper endpoint +# COMP_EVAL_NVCF_SERVICE_KEY= # allenai/wildguard safety server + # GDPVal (nemo_gym Stirrup agent) — agent web search. Secret; exported and read # by the harness. See recipes/tasks/gym/gdpval.md + references/gym-gdpval.md. # TAVILY_API_KEY= @@ -50,6 +58,13 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # /gdpval/sif. Convention: a per-user .cache dir. # GDPVAL_SIF_DIR=//.cache/gdpval/sif +# GDPVal (nemo_gym) — SIF filename inside GDPVAL_SIF_DIR. Only set this to point at a +# site-provided image whose name differs from the canonical one; it must equal the +# basename in the config's GDPVAL_CONTAINER_PATH, or NEL's `test -d` mount check passes +# and the agent silently runs code-exec unsandboxed. +# Default: python-3.13.gdpval.gym-80e4fc.sif (the GDPval-AA v2 sandbox). +# GDPVAL_SIF_NAME=python-3.13.gdpval.gym-80e4fc.sif + # GDPVal (nemo_gym) — Stirrup agent turn cap. Read at SUBMIT time from the # launching shell (the config uses ${oc.env:GDPVAL_MAX_TURNS,250}), so it must be # exported before `nel run`; setting it as a container env var has no effect. @@ -67,6 +82,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # AWS_ACCESS_KEY_ID= # AWS_SECRET_ACCESS_KEY= # NEL_NEXT_EVAL_IMAGE= # harbor eval image (arch-matched) — set by eval-config +# HARBOR_ECS_REGION= # Fargate sandbox region. Upstream defaults: us-east-1 +# # (Terminal-Bench), us-east-2 (SWE-bench). TB's ECR repo +# # NAME tracks this (harbor-); SWE-bench's does not. # HARBOR_ECR_REPOSITORY= # Terminal-Bench harbor ECR — set by eval-config -# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2) — set by eval-config +# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2, region-independent) — set by eval-config # MLFLOW_TRACKING_URI= # MLflow tracking URI (canonical frontier-evals host) — set by eval-config diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 9463c653062..1f7850f8782 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -45,17 +45,27 @@ services: proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below + model_traffic: {capture_request_body: true} # FEA-224 — set PER SERVICE, never in a shared block (an alias-only entry has no `type` and fails the service discriminator) + # Interceptor lists REPLACE wholesale on merge — if you add one, restate them all. interceptors: + # FEP-1104/1120 diagnostics — canonical position is FIRST (both TB2.1 and SWE-bench). + # Uncomment for a CANARY/debug run, drop it for the scored run: first_n caps only 200s, + # so every error pair (full req+res bodies) is retained in memory for the whole run and + # re-serialized on each write — unbounded growth exactly when the server errors. + # - name: http_pairs_dump + # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + # SWE-bench (OpenHands, multi-turn) inserts system_message + turn_counter HERE, before + # drop_params — don't lift that chain blind, see swebench_verified.md - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]} - # SWE-bench (OpenHands, multi-turn) adds turn_counter + system_message AND USES A DIFFERENT - # ORDER (drop_params before consolidate_system) — don't lift this chain — see swebench_verified.md - # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run: - # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for - # the whole run and re-serialized on each write — unbounded growth exactly when the server errors. - # - name: http_pairs_dump # canonical LAST in the chain (SWE-bench: first) - # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + - name: consolidate_system + # reasoning models only — normalize the reasoning field and replay it across turns. + # `reasoning_replay.mode` is per MODEL (think_tags default / native / both), not per + # benchmark. Drop both for instruct models. + # - name: reasoning + # - name: reasoning_replay + # config: {mode: think_tags} node_pool: gpu benchmarks: @@ -67,10 +77,12 @@ benchmarks: timeout_strategy: max # canonical TB2.1; "task" = leaderboard-comparable agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-1 # MUST match the region in ${HARBOR_ECR_REPOSITORY} (SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}) + # TB2.1's ECR repo NAME tracks the region (harbor-); SWE-bench's is pinned to + # us-west-2/harbor-swebench whatever the sandbox region is (us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}). + region: ${HARBOR_ECS_REGION:-us-east-1} ecr_repository: ${HARBOR_ECR_REPOSITORY} # from modelopttools:eval-config concurrency: 50 - log_stream_prefix: terminalbench21-??? + log_stream_prefix: terminalbench-21-??? # canonical stem + - cluster: type: slurm @@ -87,7 +99,7 @@ cluster: HF_HOME: /cache/huggingface AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} - AWS_DEFAULT_REGION: us-east-1 # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} + AWS_DEFAULT_REGION: ${HARBOR_ECS_REGION:-us-east-1} # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} LLM_API_KEY: "no-key-needed" mount_home: false auto_resume: true @@ -110,4 +122,12 @@ output: exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. - tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} + tags: + framework: vllm + model: "???" + checkpoint_path: "???" + benchmark: "???" + task_name: "terminal-bench-2.1" # upstream benchmark slug (SWE-bench: swebench-verified) + nemo-evaluator-next-version: "0.5.0.1" # eval_image version, so a score traces to its image + temperature: '1.0' + top_p: '0.95' diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml index 6e5eef250fa..91e39c144fb 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml @@ -14,7 +14,7 @@ # limitations under the License. # # ============================================================================= -# GDPVal (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode. +# GDPVal-AA v2 (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode. # Self-deploys a checkpoint via vLLM on one SLURM node. One gym eval per config. # # Read recipes/tasks/gym/gdpval.md + references/gym-gdpval.md first — they @@ -42,20 +42,25 @@ defaults: # reference deliverables and runs on the public gym image, so it works standalone. # rubric — standalone LLM-judge scoring, 0-1 reward per deliverable. (this template) # comparison — pairwise vs anchored reference deliverables; the only mode that yields -# an AA-comparable ELO / win-rate. It needs a reference set, a newer gym -# image, and its own overrides — do NOT just flip reward_mode here. +# an AA-comparable ELO, and the mode the upstream GDPVal-AA-V2 benchmark +# is defined in. It needs the 12-reference set, a py3.13 gym image, a +# 3-judge panel and multistage overrides — do NOT just flip reward_mode. # To run comparison, NVIDIA-internal users should follow `modelopttools:eval-config` # Step 3c, which converts this config (container override + reference_models map + # mounts + multistage). See references/gym-gdpval.md "Scoring modes". gdpval: reward_mode: rubric -# GDPVal pairwise judge. base_url is config (from .env), not a secret, so no -# export needed; only api_key (INFERENCE_API_KEY) is exported and read by the -# harness. Keep the judge fixed across comparable runs. +# GDPVal judge. base_url is config (from .env), not a secret, so no export needed; +# only api_key (INFERENCE_API_KEY) is exported and read by the harness. Keep the judge +# fixed across comparable runs. +# RUBRIC mode (this template) scores with this single judge. COMPARISON mode routes a +# 3-member panel — gpt-5.5 / gemini-3.1-pro / claude-opus-4.8, one SAMPLED PER TRIAL at +# judge_sampling_seed=42 — through this same proxy; the panel is added by the +# eval-config Step 3c conversion, not here. See references/gym-gdpval.md "Judge". gdpval_judge: base_url: # from .env (/v1 base); shared inference host - model: gcp/google/gemini-3.1-pro-preview # Gemini 3.1 Pro; use an equivalent on your endpoint if needed + model: gcp/google/gemini-3.1-pro-preview # rubric judge; also the panel's default member api_key: INFERENCE_API_KEY # doc only; common_params injects the VALUE ($INFERENCE_API_KEY). # An env-var NAME reaches the proxy as a literal -> opaque judge 500. @@ -136,7 +141,13 @@ evaluation: TAVILY_API_KEY: host:TAVILY_API_KEY # Stirrup agent web search UV_CACHE_DIR: lit:/cache/uv # Apptainer SIF for the Stirrup per-task code-exec sandbox (must match the mount above). - GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.sif + # gym-80e4fc is the GDPval-AA v2 sandbox: Debian trixie, python 3.13.5 asserted at + # build time, the 419 published AA package pins, and the Office metric-substitute + # fonts earlier images lacked. It is versioned WITH the Gym pin below — move both or + # neither ($SKILL_DIR/scripts/gdpval-sif.sh defaults to this pair). + GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif + # Bounds Gym's in-process rollout retry. Canonical is 3 (= two retry passes). + NEMO_GYM_MAX_ROLLOUT_ATTEMPTS: lit:3 # Shared-FS staging for ref files; node-local /tmp breaks multi-node Ray. GDPVAL_REF_FILES_DIR: lit:/gdpval_ref_files # Deliverables land under /results (already mounted) so they persist. The @@ -205,12 +216,25 @@ evaluation: # as a non-git directory, so `install_on_the_fly.commit` below is silently # ignored here (the prepare step logs "/opt/Gym is not a git repo; using baked-in # Gym version") — the pin only applies on images where /opt/Gym IS a git repo. + # ON A GIT-BACKED IMAGE THE PIN APPLIES, AND IT NEEDS PYTHON >= 3.13.14. The + # canonical config pairs it with a py3.13 gym runtime image plus + # `runtime_python: /usr/local/bin/python3.13` (uncomment below). A py3.12 image + # + this pin fails at venv creation, not at scoring. container: nvcr.io/nvidia/eval-factory/nemo-gym:26.05 # pin a verified tag nemo_evaluator_config: config: params: extra: nemo_gym: + # limit_samples is INERT on the gym path; canonical says so explicitly + # so a "2-sample smoke test" can't be believed. Also required by + # multistage (comparison mode): ++limit truncates the loaded dataset + # while stages sample from the full distribution -> 0 rollouts. + allow_limit_samples: false + # Uncomment WITH a py3.13 gym runtime image above — the pinned Gym + # declares requires-python >= 3.13.14. Move image + interpreter + pin + # as one unit. + # runtime_python: /usr/local/bin/python3.13 install_on_the_fly: url: https://github.com/NVIDIA-NeMo/Gym # Gym version. BUMPING THIS REQUIRES REBUILDING THE SIF from the matching @@ -218,15 +242,17 @@ evaluation: # old SIF + new gym silently degrades deliverables. Rebuild with # `gdpval-sif.sh --commit ` to a new name + repoint GDPVAL_CONTAINER_PATH. # See references/gym-gdpval.md "Rebuild the SIF when the Gym version changes". - # Current golden pin (updated GDPVal task-sampling algo, which drives - # multistage stage-1 selection). Its gdpval.def is byte-identical to - # 049b1fd0…, so a python-3.13 SIF built from either commit is valid — - # no rebuild when moving between them. + # Current canonical pin (GDPval-AA v2). Carries the AA reference-ELO + # anchor refresh, the three multistage reference anchors, long-context + # budgeting (completion floor 8192), and the gdpval.def the gym-80e4fc + # SIF above was built from. Runs on it are NOT strictly comparable to + # runs on the older dd41196f pin — the agent's prompt and generation + # behaviour both changed. # NOTE: this pin is INERT on the public nemo-gym image (see the # `container:` comment above) — it only takes effect on an image where # /opt/Gym is a git repo. Confirm via "=== NeMo Gym commit ===" + SHA # in the client log before crediting it with any behaviour change. - commit: dd41196f620f2af99947d776cbe5da9439d2a08d # pragma: allowlist secret + commit: df3e201d942f6397def1ec8c10037d29115fcbbd # pragma: allowlist secret command: | set -ex cd /opt/Gym @@ -327,6 +353,9 @@ evaluation: # No judge_responses_create_params_overrides.model here: the judge model is # already set by openai_model=${gdpval_judge.model}, and pinning it a second # way silently collapses comparison mode's 3-member judge panel to one judge. + # Comparison mode additionally appends num_comparison_trials=4, + # judge_sampling_seed=42, the judge_panel list, the reference_models map and + # the multistage block — all from eval-config Step 3c, none of them here. # num_repeats=1 — both current goldens. No reference_* overrides here: # the single-reference keys conflict with comparison mode's reference_models # map (added by modelopttools:eval-config Step 3c). @@ -357,7 +386,7 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | GDPVal rubric | T=1.0, top_p=0.95, num_repeats=1' + description: 'CHANGEME-served-model-name | GDPVal-AA v2 rubric | T=1.0, top_p=0.95, num_repeats=1' log_logs: true only_required: false tags: diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 01c1c9a42c4..1b2d4fe8267 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -86,9 +86,11 @@ deployment: gpu_memory_utilization: 0.95 # golden raises this from 0.85 to fit the 1M KV cache # Golden serves in REASONING mode (thinking forced on via chat_template_kwargs # below), so set the model's --reasoning-parser. - # After sizing num_nodes / num_instances / TP / DP above, append - # `--max-num-seqs N` where N = ceil(parallelism / num_instances / DP) — the GYM - # rule, not the generic SKILL one. Golden: ceil(256/4/2) = 32. + # After sizing num_nodes / num_instances / TP / DP above, append `--max-num-seqs N`. + # Do NOT derive N from `parallelism`: the canonical bench.yaml now sets parallelism + # 512 DELIBERATELY ABOVE total server capacity so 1M-token requests queue at the + # endpoint instead of leaving replicas idle. Size N from what a replica's KV can hold + # (the golden ran 32 per replica × 8 replicas = 256 in flight against parallelism 512). # Treat that as a CEILING, not a target. Like AA-LCR (recipes/tasks/aa/lcr.md) # this task is KV-bound, and far more so: ~1M input tokens per request vs LCR's # ~120K. Too high and vLLM preempts, and recomputing a 1M-token prefill makes the @@ -131,10 +133,16 @@ evaluation: params: temperature: 1.0 top_p: 0.95 - # Gym client concurrency, TOTAL across instances — not a per-server cap. - parallelism: 256 + # Gym client concurrency, TOTAL across instances — not a per-server cap, and + # intentionally ABOVE it: the canonical bench.yaml uses 512 so the endpoint + # always has queued work while Gym's own dispatch stays bounded. Drives both + # ++num_samples_in_parallel and ++global_aiohttp_connector_limit_per_host. + parallelism: 512 request_timeout: 36000 # 1M-token prefills are slow max_retries: 10 + # null = full run. Set N for a canary: unlike GDPVal, MRCR forwards it to the + # gym as ++limit (see the gated line in collect_rollout_params below). + limit_samples: # Uncapped: answers reproduce a full earlier turn, and any cap truncates # it and craters SequenceMatcher.ratio (see max_output_tokens=null below). max_new_tokens: @@ -292,9 +300,12 @@ evaluation: ++reuse_existing_data_preparation=true ++skip_venv_if_present=true ++upload_rollouts_to_wandb=false + ++observability_enabled=true + ++model_call_capture_dir={{config.output_dir}}/model_calls ++responses_create_params.temperature={{config.params.temperature}} ++responses_create_params.top_p={{config.params.top_p}} ++responses_create_params.max_output_tokens=null + {% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %} common_params: >- ++num_repeats=1 ++policy_base_url={{target.api_endpoint.url}} diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index bbd70551cb9..289cf7c672d 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -8,6 +8,11 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. > **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in > nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image > pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run. +> +> That dir also ships `bench_direct.yaml`, a **different backend**: the Gym/opencode +> route (`++num_samples_in_parallel=1024`, its own Gym commit pin, opensandbox) used by +> the P/D recipes. `benchmark_backend: evaluator` — the harbor path described here — is +> what this skill runs. Don't mix values between the two. ## Task-specific values (canonical `bench.yaml`) @@ -16,12 +21,15 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | scope | 500 Python tasks × `repeats: 5` | -| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | +| `max_concurrent` / `sandbox.concurrency` | `15` (canonical `bench.yaml`); the reviewed per-model leaves keep it. A separate customer-launch track (`reviewed_configs/clu/…`) runs `repeats: 3` / `concurrency: 10` — **not** AA-comparable, don't borrow its numbers | | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | -| `sandbox.region` | `us-east-2` | -| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-2}` (differs from TB2.1's `us-east-1` default) | +| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` → `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench`. **Hardcoded us-west-2 upstream** — unlike TB2.1 the repo does *not* track `HARBOR_ECS_REGION`; moving the sandbox region leaves the ECR alone | | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) *(shared — see `references/nel-next.md`)* | -| `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | +| `cluster.container_env.AWS_DEFAULT_REGION` | `${HARBOR_ECS_REGION:-us-east-2}` (match `sandbox.region`) | +| `sandbox.log_stream_prefix` | canonical stem `swebench---` | +| `output.export_config.mlflow.tags` | canonical adds `task_name: swebench-verified` and `nemo-evaluator-next-version: 0.5.0.1` | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block | | `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) | | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | @@ -32,7 +40,7 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. benchmarks: - playbook: swebench_verified repeats: 5 - max_concurrent: 15 # keep == sandbox.concurrency; per-model configs may raise both + max_concurrent: 15 # canonical; keep == sandbox.concurrency instruction_template: /configs/prompts/swebench_instruction.md # mounted (see gotcha) solver: service: @@ -40,7 +48,7 @@ benchmarks: run_timeout: 10800 agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-2 + region: ${HARBOR_ECS_REGION:-us-east-2} # canonical; the ECR below stays us-west-2 ecr_repository: ${HARBOR_SWEBENCH_ECR_REPOSITORY} concurrency: 15 log_stream_prefix: swebench-verified-- @@ -88,10 +96,16 @@ OpenHands runs ~200 turns/task. The canonical config adds a `system_message` interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) plus `turn_counter`. -**Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` -comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it -retains every error pair in memory for the whole run (`references/nel-next.md`); drop it -from the scored config. +**Order.** `http_pairs_dump` is **first**, then `system_message` → `turn_counter` → +`drop_params` → `consolidate_system` → `reasoning` → `reasoning_replay`. (TB2.1 now puts +`http_pairs_dump` first too; the SWE-bench-only part is the `system_message` + +`turn_counter` pair.) `http_pairs_dump` is canary/diagnostic-only — it retains every error +pair in memory for the whole run (`references/nel-next.md`); drop it from the scored config. + +**Interceptor lists replace wholesale on merge.** The reviewed leaf restates the model +fragment's whole chain to add two entries — it does not append. Adding `system_message` +without re-listing `drop_params`/`consolidate_system`/`reasoning`/`reasoning_replay` +silently drops them. ```yaml proxy: @@ -127,11 +141,21 @@ Fargate quota and `N × gpus_per_node` against your allocation. ## Score Extraction -Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): +Report **`pass@1`** — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): the resolved rate over the 500 tasks, **already averaged over repeats** (nel-next reports a single `pass@1`; there is **no `avg-of-N` key** like the 0.2.6 nemo-skills -metrics). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer -table) in the run dir or `nel eval report -r `, then push to MLflow with +metrics). MLflow logs it as `pass_at_1`. + +> The upstream `manifest.yaml` lists its metric key as **`mean/reward`**, not `pass@1` — +> that is the key the Gym/direct (`bench_direct.yaml`) recipes emit, and it is shared by +> the whole SWE-bench family (Verified / Multilingual / Pro). On the harbor path the +> certificates in `reviewed_configs/…/swebench_verified/certificates/` record `pass@1` +> (e.g. `0.516`), so `pass@1` is still what you report for a nel-next run. Quote the key +> name alongside the number so a `mean/reward` from the direct path is never compared +> against it. + +Read from `report.md` (Benchmark / Scorer table) in the run dir or +`nel eval report -r `, then push to MLflow with `nel-next.sh mlflow-push -r -c ` (SLURM doesn't auto-export). Keep `timeout_strategy` + the instruction/system prompt fixed across baseline vs quantized for a valid delta. diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index 42a5366735a..b022d74c498 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -18,23 +18,35 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | agent | `terminus-2` (from the playbook) | | `repeats` | `8` (AA / leaderboard count — don't lower for a scored run) | | sandbox | `ecs_fargate`, `stateful: true` (agent + verifier share one container) | -| `sandbox.region` | match the region in `${HARBOR_ECR_REPOSITORY}` (us-east-1 with the eval-config default) | -| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by the `modelopttools:eval-config` skill (internal harbor ECR) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-1}` — one env var now drives region **and** repo name | +| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by `modelopttools:eval-config`; canonical value is `463701203462.dkr.ecr..amazonaws.com/harbor-`, i.e. the repo **name tracks the region** (`harbor-us-east-1`). Changing region means changing both halves. | | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | +| `sandbox.log_stream_prefix` | canonical stem `terminalbench-21`; the reviewed leaf appends model + framework (`terminalbench-21-nano-v35-vllm`) | | `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) | | timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable | | `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) *(shared — see `references/nel-next.md`)* | | `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` *(shared — see `references/nel-next.md`)* | -| `http_pairs_dump` | **last** in the interceptor chain — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `http_pairs_dump` | **first** in the interceptor chain (same position as SWE-bench) — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block, which would inject an alias-only entry with no `type` into other models' composed configs | +| `output.export_config.mlflow.tags` | canonical adds `task_name: terminal-bench-2.1` and `nemo-evaluator-next-version: 0.5.0.1` alongside `model`/`framework` | | scope | 89 tasks × `repeats: 8` | These values mirror the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in `configs/shared/nel_next_containers.yaml`. See `references/nel-next.md` + the eval-config -"source of truth" note. The `benchmarks:` block (drop into the example template): +"source of truth" note. + +> **`bench_direct.yaml` is a different backend, not an alternative spelling.** The same +> upstream dir now also ships a Gym-native route (Terminus 2 agent + opensandbox, +> `++num_samples_in_parallel=512`, its own Gym commit pin) used by the P/D recipes. It +> scores the same 2.1 task set but through different machinery; `benchmark_backend: +> evaluator` — the harbor path below — is what this skill runs and what the nel-next +> deltas here describe. Don't mix values between the two. + +The `benchmarks:` block (drop into the example template): ```yaml benchmarks: @@ -49,10 +61,10 @@ benchmarks: llm_kwargs: timeout: 3600 # per-request LLM timeout (canonical) sandbox: - region: us-east-1 # must match the region in ${HARBOR_ECR_REPOSITORY} + region: ${HARBOR_ECS_REGION:-us-east-1} # canonical; repo name below must track it ecr_repository: ${HARBOR_ECR_REPOSITORY} # from eval-config (internal harbor account/region) concurrency: 50 - log_stream_prefix: terminalbench21-- + log_stream_prefix: terminalbench-21-- ``` `cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor`) and the AWS creds @@ -66,7 +78,8 @@ it is purely a wall-clock lever. `shards: 4` suits 89 × r8 = 712 trials. Check ## Score Extraction -Report **`pass@1`** only — benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): +Report **`pass@1`** — the key the upstream `manifest.yaml` certifies on, and the only +one to quote. Benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): the resolved rate over the 2.1 task set, **already averaged over repeats** (a single `pass@1`; no `avg-of-N` key). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer table) or `nel eval report -r `, then push to MLflow with diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md index c8f581995a1..2208aaf7141 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md @@ -1,16 +1,26 @@ -# GDPVal (NeMo Gym "Stirrup" agent) +# GDPVal-AA v2 (NeMo Gym "Stirrup" agent) ## Task Details - Reference: `references/gym-gdpval.md` (SIF build, gym machinery, deploy sizing, - scoring modes, failure modes) — **read it before editing a GDPVal config.** + scoring modes, reference ladder, multistage, failure modes) — **read it before + editing a GDPVal config.** +- Upstream config: `configs/benchmarks/gdpval-aa-v2/` → `configs/benchmarks/gym/gdpval/` + (`bench.yaml` + `refs.yaml`) in nvidia-eval-factory-benchmarking + (`dl/JoC/competitive_evaluation/…`) — the source of truth; re-read it before a scored run. - Upstream README: GDPVal is an **agentic** benchmark: the Stirrup agent produces office/PDF -deliverables inside a per-task Apptainer code-exec sandbox, then a pairwise/rubric -judge (**Gemini 3.1 Pro**) scores them. It is the most resource-intensive benchmark -in the suite — **220 tasks**, `num_repeats=1`, 4 judge trials per rollout. +deliverables inside a per-task Apptainer code-exec sandbox, then a judge scores +them — one rubric judge, or in comparison mode a **3-member panel** (gpt-5.5 / +gemini-3.1-pro / claude-opus-4.8) sampled per trial. It is the most +resource-intensive benchmark in the suite — **220 tasks**, `num_repeats=1`, 4 +comparison trials per rollout. + +**Upstream is now GDPVal-AA v2** — a rebuilt sandbox matching the published AA +environment, a refreshed 12-reference ELO ladder, and a two-stage anchored fit. A v2 +score is not comparable to a v1 one. It is currently validated with the **0.2.6 `nel` launcher** as a `nemo_gym` task (NOT nel-next), so Steps 1–9 apply — but with the branch differences below. @@ -33,14 +43,19 @@ suite membership, so read that per-task, not from the path. - **Apptainer SIF sandbox** — prefer a site-provided SIF; otherwise `$SKILL_DIR/scripts/gdpval-sif.sh` builds one into `$GDPVAL_SIF_DIR` (build-if-absent, never copied between clusters). Missing/misnamed → **silent** unsandboxed exec. + Canonical is `python-3.13.gdpval.gym-80e4fc.sif`, and it is versioned **with** the Gym + pin and the eval image's Python — see the reference's "Three things that move together". - **Thinking mode is mandatory** — non-thinking loses ~86% of pairwise judgements. Serve with the model's `--reasoning-parser` and force it on via the adapter's `chat_template_kwargs`. - **Scoring:** `rubric` (template default, no references, no ELO) vs `comparison` - (the AA-comparable `normalized_elo`; a conversion, not a flag flip). + (the AA-comparable `normalized_elo`; a conversion, not a flag flip). Comparison adds + the 12-reference ladder and a **two-stage** fit: stage 1 samples 45 tasks against all + references, stage 2 judges all 220 against the 4 nearest the stage-1 estimate. - Needs `INFERENCE_API_KEY`, `TAVILY_API_KEY`, `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR` in `.env`, plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the config has a - `pre_cmd`). + `pre_cmd`). Comparison mode additionally wants `JUDGE_API_KEY` and the auxiliary + endpoint keys — see the reference's "Env vars". All of the above — SIF handling, the SIF↔Gym-commit coupling, scoring modes, judge panel, preflight and failure modes — is detailed in **`references/gym-gdpval.md`**. @@ -64,7 +79,12 @@ current goldens use it. A full 220-task run of a large MoE typically needs multi **`++…params.limit_samples=N` is inert on the gym path.** The gym does its own data prep and rollout collection, so the launcher-level limiter is ignored: you get the full 220-task run. Do not use it believing you launched a two-task smoke test — this -is the heaviest benchmark in the suite. +is the heaviest benchmark in the suite. The canonical config states this in the config +itself (`extra.nemo_gym.allow_limit_samples: false`), and the template carries it too. + +**`++limit` is worse than useless in comparison mode**: it truncates the loaded dataset +while the stages sample from the full distribution, so you get **0 rollouts**. (MRCR is +the opposite — there `limit_samples` *does* reach the gym. Don't carry the habit across.) There is no cheap sample-limited canary. Instead, **launch the real run and treat its first ~20–30 minutes as the canary**, cancelling if any of these is wrong: @@ -77,8 +97,12 @@ grep -ciE " 401 | 403 |Internal Server Error" $RD/artifacts/nemo_gym_logs/gdpval wc -l $RD/artifacts/evaluator_rollouts.jsonl # rollouts flowing ``` -In comparison mode stage 1 (45 tasks) is a natural early checkpoint — an ELO estimate -appears before the full 220-task stage 2 starts. +In comparison mode stage 1 (45 tasks, all references) is a natural early checkpoint — an +ELO estimate appears before the full 220-task stage 2 starts. Stage 1 runs with +`partial_completion` gates (≥0.9 overall success, ≥0.5 per reference, ≥1 judged row per +reference; `timeout_exceeded` and `transient` waivable) so one reaped rollout or one judge +400 no longer forfeits stage 2 — if you see stage 2 refuse to start, check those gates +before blaming the model. ## Score Extraction @@ -88,29 +112,37 @@ appears before the full 220-task stage 2 starts. **The reported GDPVal score is `normalized_elo`** — the AA 0–1 scale, comparable across models and to the published AA index. `eval_elo` is the same fit on the raw -Elo axis (`normalized_elo = (eval_elo - 500) / 2000`); quote it as supporting -detail, not as the score. +Elo axis (`normalized_elo = (eval_elo - 500) / 2000`, fitted across all models); quote it +as supporting detail, not as the score. + +**Read it from the FINAL STAGE key, not the flat one.** The upstream manifest certifies +on `comparison/stage_1/normalized_elo` (stage index 1 = the 220-task stage), because the +flat `comparison/normalized_elo` is emitted **only when that stage is exactly complete** — +a few lost rollouts blank it out while the stage-scoped key still carries the fit. A +missing flat key is therefore not evidence the run failed to score. The final numbers live in **`artifacts/results.yml`** (authoritative, local) and are mirrored to MLflow. Read them by metric name: | Mode | Metric (results.yml → `groups.nemo_gym.metrics..scores..value`) | | --- | --- | -| comparison | `gdpval_stirrup_agent/comparison/normalized_elo` ← **REPORT THIS** (AA 0–1 scale) | -| comparison | `gdpval_stirrup_agent/comparison/eval_elo` (raw Elo; supporting detail) | +| comparison | `gdpval_stirrup_agent/comparison/stage_1/normalized_elo` ← **REPORT THIS** (AA 0–1 scale, final stage) | +| comparison | `gdpval_stirrup_agent/comparison/stage_1/eval_elo` (raw Elo; supporting detail) | +| comparison | `gdpval_stirrup_agent/comparison/stage_0/eval_elo` — the stage-1 (45-task) estimate; an early checkpoint, **not** the score | +| comparison | flat `gdpval_stirrup_agent/comparison/{normalized_elo,eval_elo}` — same quantity, but present only on an exactly-complete final stage | | comparison | `gdpval_stirrup_agent/comparison/win_rate`, `/judged`, `/wins`, `/losses`, `/ties` | | comparison | per-reference: `gdpval_stirrup_agent/comparison/ref//{win_rate,wins,losses,ties,judged}` | -| comparison | per-stage estimate: `gdpval_stirrup_agent/comparison/stage_0/eval_elo` (stage 1, all refs) — the **final** value is the top-level one, from the last stage | | rubric | mean of `reward` across `artifacts/evaluator_rollouts.jsonl` (per-rollout 0–1) | ```bash -# COMPARISON mode — final score from the local results file (no MLflow needed) +# COMPARISON mode — final score from the local results file (no MLflow needed). +# stage_1 first (what upstream certifies on); the flat key is the fallback. python3 -c " import yaml m=yaml.safe_load(open('//nemo_gym.0/artifacts/results.yml'))['groups']['nemo_gym']['metrics'] -for k in ('normalized_elo','eval_elo','win_rate'): +for k in ('stage_1/normalized_elo','stage_1/eval_elo','normalized_elo','eval_elo','win_rate'): n=f'gdpval_stirrup_agent/comparison/{k}' - print(k, '=', m[n]['scores'][n]['value'])" + if n in m: print(k, '=', m[n]['scores'][n]['value'])" # RUBRIC mode (the template default) — there is no ELO; average the per-rollout reward python3 -c " @@ -126,8 +158,9 @@ comparison run logs **~200 metrics and most of them are per-reference**, so the headline is easy to miss: ```text -nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/normalized_elo <- report this -nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/eval_elo +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/stage_1/normalized_elo <- report this +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/stage_1/eval_elo +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/normalized_elo <- flat fallback nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/win_rate ``` diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md index 2123d1c31c1..fd8a3f08040 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md @@ -5,6 +5,9 @@ - Benchmark: - Resource server: - Dataset: `openai/mrcr` (HF, gated → `HF_TOKEN`) +- **Source of truth:** `configs/benchmarks/mrcr/bench.yaml` + `manifest.yaml` (1M) and + `configs/benchmarks/mrcr-128k/` (128K) in nvidia-eval-factory-benchmarking + (`dl/JoC/competitive_evaluation/…`) — re-check before a scored run. Long-context retrieval. Each task is a long multi-turn conversation with N near-identical "needle" responses; the model must reproduce the Nth verbatim @@ -60,6 +63,21 @@ The n3 variants drop over-long samples, so all three are different datasets and candidate, and set it in **both** `data_prep_params` and `collect_rollout_params` — changing one prepares one dataset and rolls out another. +The metric prefix column is the **manifest metric key**: upstream certifies 1M on +`mrcr_n3_1m_benchmark_simple_agent/pass@1/accuracy` and 128K on +`mrcr_n3_128k_benchmark_simple_agent/pass@1/accuracy`. + +**128K is now its own upstream benchmark, on a newer launcher schema.** +`configs/benchmarks/mrcr-128k/` is a separate slug with its own manifest, and its +`bench.yaml` uses the condensed gym block — `benchmark: mrcr/config_n3_128k`, +`runtime_dependencies: tiktoken transformers`, `prepare_args` / `run_args` — against the +`gym_runtime` container, instead of this template's `data_prep_params` / +`collect_rollout_params` / inline `command:` bootstrap. Both reach the same Gym pin +(`a431501a`); the condensed form just moves the bootstrap into the runtime image. This +template stays on the explicit form, which is what the 1M `bench.yaml` still ships. +If you switch a config to the condensed schema, switch the container with it — the +`pre_cmd` and `command:` blocks here have no effect there. + The `num_repeats` column is what each variant **declares upstream**, not necessarily what runs: for `type: benchmark` datasets the value is a placeholder and the runner decides. The template therefore pins `++num_repeats=1` in @@ -89,10 +107,16 @@ deliberately. **Do not change repeat counts when aligning to a golden.** - Fan out via `execution.num_nodes` / `num_instances` (HAProxy pattern A — `references/multi-node.md`). **Size these from the cluster's GPUs-per-node**, do not copy: pick TP for the model, fill the node with DP, then choose instances for - the replica count you want. `parallelism` is the total across instances, so - `--max-num-seqs = ceil(parallelism / num_instances / DP)`. The golden ran 4 nodes - × (TP2 × DP2) on 4-GPU nodes = 8 replicas, `ceil(256/4/2) = 32` each; the same 8 - replicas on 8-GPU nodes is 2 × (TP2 × DP4). + the replica count you want. The golden ran 4 nodes × (TP2 × DP2) on 4-GPU nodes = + 8 replicas; the same 8 replicas on 8-GPU nodes is 2 × (TP2 × DP4). +- **`parallelism` is 512 and is NOT a server cap — do not derive `--max-num-seqs` + from it.** The canonical `bench.yaml` raised it from 256 with the comment that MRCR + prompts run to 1M tokens, "so allow the endpoint to queue requests while keeping + Gym's client concurrency bounded". It sets `++num_samples_in_parallel` and + `++global_aiohttp_connector_limit_per_host`; the server's in-flight cap is + independent and deliberately smaller (the golden's 8 replicas × `--max-num-seqs 32` + = 256, half of 512 — the rest queue). The old `ceil(parallelism / num_instances / DP)` + identity was a coincidence at 256 and is wrong at 512. - **`--max-num-seqs` is a ceiling, not a target.** MRCR is the most KV-bound task in the skill — ~1M input tokens per request against AA-LCR's ~120K — so AA-LCR's rule applies harder: oversubscribe and vLLM preempts, and recomputing a 1M-token @@ -139,28 +163,28 @@ NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image. ## Canary -MRCR's gym path takes `++limit=N` (the launcher-level `limit_samples` does not -reach the gym). **Not verified on this pinned commit** — treat the first ~30 min -of the real run as the canary, as the GDPVal recipe does. `++limit` caps rollouts -only: the 1M tokenize/drop-over-long **prepare pass still runs in full**, so a -5-sample canary is not cheap. Append it to `collect_rollout_params` in your copy -of the YAML — easier than re-pasting the whole folded scalar through `-o`: - -```yaml - collect_rollout_params: >- - ... - ++limit=5 # canary only — remove for the scored run -``` +**`limit_samples` now reaches the gym.** The canonical `bench.yaml` gates the override +on it — `{% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %}` +— and the template carries the same line, so `-o ++config.params.limit_samples=5` (or +editing `limit_samples:` in your copy) produces `++limit=5`. This is the opposite of +GDPVal, where the launcher-level limiter is inert. -Then watch the first ~30 min of the real run: +`++limit` caps **rollouts only**: the 1M tokenize/drop-over-long **prepare pass still +runs in full**, so a 5-sample canary is not cheap, and `pass@1` from a limited run is +not a score. Also watch the first ~30 min of the real run: ```bash RD=//nemo_gym.0 grep -A1 "=== NeMo Gym commit ===" $RD/logs/client-*.log | grep -c a431501a # pin applied grep -ciE "ModuleNotFoundError|tiktoken" $RD/logs/client-*.log # pre_cmd didn't take wc -l $RD/artifacts/evaluator_rollouts.jsonl # rollouts flowing +ls $RD/artifacts/model_calls | head # observability capture on ``` +`++observability_enabled=true` + `++model_call_capture_dir` are canonical and write +per-call records under `artifacts/model_calls` — useful for diagnosing the prefix-gate +failure below. They grow with the run; drop them only if you are short on space. + **Preempted vs timed out.** A 1M run routinely exceeds 4h. `TIMEOUT` auto-resumes from the response cache; `CANCELLED by ` (preemption) does not — its chained job exits in ~20s (`…finished with 'CANCELLED…' state. EXIT!`), which is expected, @@ -202,8 +226,11 @@ flat. Before quoting, check truncation: `eval_factory_metrics.json` → at `--max-model-len 1100000`). Reference shape (reviewed golden, BF16 Nano 3.5, 1M): `pass@1 = 26.91` (2/4/8 -needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, parallelism 256, 4 nodes / -4 instances. Use it to sanity-check shape, not as a bar for another model — a +needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, 0 failures, 4 nodes / 4 +instances, Gym `a431501a`, toolchain `0.2.7+20260722.eb3ddf2a` — the sign-off run +recorded in the upstream `manifest.yaml`. That run used `parallelism: 256`; the +canonical `bench.yaml` has since moved to 512, which is a throughput change, not a +scoring one. Use the shape to sanity-check, not as a bar for another model — a rollout count well below 2363 (full runs only — a `++limit` canary is expected to be short) means tasks were lost (e.g. a walltime resume) and the score covers fewer tasks than the reference. diff --git a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md index f935e50ef57..b8ba53f2eba 100644 --- a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md +++ b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md @@ -3,14 +3,23 @@ GDPVal is currently validated with the **0.2.6 `nel` launcher** as a `nemo_gym` task, but it is mechanically unlike the `aa/` nemo-skills tasks: the Stirrup agent produces office/PDF **deliverables** in a per-task **Apptainer** code-exec sandbox, -a pairwise/rubric **judge** (Gemini 3.1 Pro) scores them, and NeMo Gym is pulled -and run **inline in the eval container** (`install_on_the_fly`) via +a rubric judge or a **3-member pairwise judge panel** scores them, and NeMo Gym is +pulled and run **inline in the eval container** (`install_on_the_fly`) via `ng_prepare_benchmark` + `ng_e2e_collect_rollouts`. This file is the shared machinery; the config template is `recipes/examples/gym/example_gdpval.yaml` and the per-task pointer is `recipes/tasks/gym/gdpval.md`. The gym bootstrap machinery described here is shared with MRCR (`recipes/tasks/gym/mrcr.md`) — fixes here apply to both examples. +**Upstream this benchmark is `GDPVal-AA-V2`** (`configs/benchmarks/gdpval-aa-v2/` → +`configs/benchmarks/gym/gdpval/{bench,refs}.yaml` in nvidia-eval-factory-benchmarking, +`dl/JoC/competitive_evaluation/…`). v2 is a different measurement from the v1 configs +this skill previously tracked: a rebuilt sandbox aligned to the **published AA GDPval-AA +v2 environment**, a refreshed 12-model reference ladder, an anchored multi-stage ELO fit, +and stage-scoped metric keys. Do not compare a v2 number to a v1 one. Re-read the +upstream `bench.yaml` + `refs.yaml` before any scored run — the reference ELO anchors +move with the AA board. + Always invoke GDPVal through the pinned wrapper, even if `nel` is already on PATH: ```bash @@ -59,15 +68,50 @@ versions within a baseline-versus-candidate comparison. | Policy model (under test) | your self-deployed vLLM endpoint (SLURM GPU node) — or an external endpoint | | NeMo Gym + Stirrup agent orchestration | inside the **eval** container (`nemo_gym` task), pulled via `install_on_the_fly` | | Per-task code-exec | **Apptainer SIF** launched by the agent inside the eval container | -| Judge (pairwise/rubric) | external OpenAI-compatible endpoint (`gdpval_judge`, e.g. Gemini 3.1 Pro) | +| Judge (pairwise/rubric) | external OpenAI-compatible endpoint (`gdpval_judge`); one model in rubric mode, a sampled 3-member panel in comparison mode | | Agent web search | Tavily (`TAVILY_API_KEY`) | +The canonical comparison-mode config also reaches two **auxiliary** endpoints through the +same proxy pattern — an `allenai/wildguard` safety server and a `Qwen3-235B` helper — each +with its own key (`COMP_EVAL_NVCF_SERVICE_KEY`, `INTEGRATE_NVAPI_KEY`). They are part of the +eval-config Step 3c conversion, not of the rubric template. + +## Three things that move together + +The Gym pin, the eval image's Python, and the SIF are **one unit**. The canonical pin +`df3e201d942f6397def1ec8c10037d29115fcbbd` declares `requires-python >= 3.13.14`, so it +must run on a py3.13 gym runtime image with +`extra.nemo_gym.runtime_python: /usr/local/bin/python3.13`, and it must mount the SIF +built from *its own* `gdpval.def` (`python-3.13.gdpval.gym-80e4fc.sif`). Move one and you +must move all three: + +| | Canonical (GDPval-AA v2) | +|---|---| +| Gym `install_on_the_fly.commit` | `df3e201d942f6397def1ec8c10037d29115fcbbd` | +| eval container | a py3.13 gym runtime image (internal `gym_runtime_py313`, CPython 3.13.14) | +| `runtime_python` | `/usr/local/bin/python3.13` — overrides the model common's py3.12 default | +| SIF | `python-3.13.gdpval.gym-80e4fc.sif` | + +A py3.12 image with this pin fails at sub-venv creation (loud). An old SIF with this pin +degrades deliverables inside the sandbox (silent). Those are the two failure shapes. + ## Apptainer SIF sandbox The Stirrup agent runs each task's generated code in an Apptainer SIF, bind-mounted at **exactly** the path `GDPVAL_CONTAINER_PATH` names (template: -`/gdpval/sif/python-3.13.gdpval.sif`). Missing or misnamed → the agent **silently** -runs code-exec unsandboxed; the run "succeeds" but the numbers aren't comparable. +`/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif`). Missing or misnamed → the agent +**silently** runs code-exec unsandboxed; the run "succeeds" but the numbers aren't +comparable. + +**What gym-80e4fc is.** The first sandbox aligned to the environment AA publishes for +GDPval-AA v2: Debian trixie (not the `python:3.13` docker image, whose 3.13.15 differs +from the closure's pinned 3.13.5 — the def *asserts* 3.13.5 at build time and fails +otherwise), the 419 published Python pins installed verbatim from the vendored freeze, +the 762-package apt closure, and the Calibri/Cambria **metric-substitute fonts** earlier +images lacked — whose absence had been reflowing every deliverable written in those +fonts. It deliberately ships **no** torch/keras/jax; the agent prompt used to advertise +them and no longer does. The name carries no architecture: build natively per cluster and +read the `BUILD_INFO` file beside a site-provided image before copying one anywhere. **If your site provides a prebuilt SIF, use it** — a self-built one resolves its pip stack at *your* build time and can drift from the sandbox a published reference set @@ -81,17 +125,22 @@ srun -p cpu -t 01:00:00 --pty "$SKILL_DIR/scripts/gdpval-sif.sh" # uses $GDPVA ``` `gdpval-sif.sh` is idempotent (flock-guarded, atomic): it builds from `gdpval.def` at -the pinned commit if absent and is a no-op once present. It needs +the pinned commit if absent and is a no-op once present. The v2 def declares a +**`%files` section** (the pin list, the apt closure, the arm64 exclusions, the sandbox +verifier), and apptainer resolves those sources against the *build's CWD* — so the def +can no longer be built on its own. The script stages the def plus every source named in +its `%files` block into a temp dir and builds from inside it; a def that adds a file +keeps working because the list is parsed, not hardcoded. It needs apptainer/singularity with unprivileged-build support and network egress — run it on a login or CPU node, **not** inside the eval job. The eval image doesn't ship apptainer, so the config's `pre_cmd` installs the **runtime** (arch-aware: use the Ubuntu PPA, not an amd64 `.deb` — most Blackwell/Grace clusters are aarch64), which needs `NEMO_EVALUATOR_TRUST_PRE_CMD=1`. -**The SIF is versioned with the Gym repo.** `gdpval.def` changes across commits (e.g. -`2502893977` → `049b1fd0` moved python 3.12 → 3.13 and added TeX Live, playwright, -polars, geospatial), and the newer agent's prompt advertises that richer runtime. So -when you bump `install_on_the_fly.commit`, **diff the def at the two commits** +**The SIF is versioned with the Gym repo.** `gdpval.def` changes across commits (the +v1→v2 move rebuilt it wholesale onto the published AA closure; an earlier one moved +python 3.12 → 3.13), and the agent's prompt describes whatever runtime the def +installs. So when you bump `install_on_the_fly.commit`, **diff the def at the two commits** (`raw.githubusercontent.com/NVIDIA-NeMo/Gym//responses_api_agents/stirrup_agent/containers/gdpval.def`); if it changed, rebuild to a **new version-tagged filename** (`GDPVAL_SIF_NAME=… gdpval-sif.sh --commit `) and repoint `GDPVAL_CONTAINER_PATH`. Running a new gym on an old SIF makes the @@ -158,21 +207,85 @@ self-deploys single-node vLLM, which is fine for a canary or a small policy. For + **`rubric`** (template default) — judge scores each deliverable against its rubric. 0–1 reward, **no ELO** (undefined without an opponent). Runs on the public gym image. + **`comparison`** — pairwise vs anchored reference deliverables; the **only** mode - yielding the AA-comparable `normalized_elo`. It is a conversion, not a flag flip: - it needs a reference set, a gym image whose Gym has the `reference_models` map, ref - mounts on **both** deployment and evaluation, and multistage overrides. Setting + yielding the AA-comparable `normalized_elo`, and the mode GDPVal-AA-V2 is defined in + upstream. It is a conversion, not a flag flip: it needs the reference set, a py3.13 + gym image whose Gym has the `reference_models` map, ref mounts on **both** deployment + and evaluation, the judge panel, and the multistage block. Setting `reward_mode=comparison` alone exits at startup with `reward_mode=comparison requires reference_deliverables_dir to be set`, surfaced only as `Process gdpval_resources_server finished unexpectedly!`. NVIDIA-internal: `modelopttools:eval-config` Step 3c is the conversion checklist. +### Comparison mode: the reference ladder + +`refs.yaml` is the single source of truth — each reference is one stanza (ELO anchor + +deliverables dir), and both the mounts and the `++…reference_models.*` overrides are +derived from it, so a path or anchor is edited in exactly one place. + +**12 active references**, spanning ELO 274 → 1468 (deepseek_v4_flash_0731 1468, +deepseek_v4_pro 1223, inkling_small 1191, glm51_fp8 1181, kimi_k26 1115, +nemotron3_ultra 1091, qwen36_35b 992, qwen35_122b 925, qwen35_397b 905, gptoss_120b 745, +gemma4_26b 713, qwen3_30b_thinking 274). Two are **deliberately disabled** and must not be +re-enabled casually: + ++ `human_gold` — 35/220 tasks ship no gold deliverable upstream, which the judge scores + as free candidate wins, and the panel rates gold ~782 against a 1000 anchor. ++ `gptoss_20b` — 23% empty deliverables, same free-win problem. + +**Anchors are held FIXED by the Bradley-Terry fit, so a stale anchor shifts every +candidate score by roughly the anchor error.** The current set was read off the AA live +board on 2026-09-09 (2026-09-11 for the three additions); the previous set sat 30–76 +points high. Match the model **variant and collection date**, not just the family — AA +scores e.g. "DeepSeek V4 Pro (Reasoning, Max Effort)" separately from "DeepSeek V4 Pro +0813". Adding a reference means staging its deliverables under the refs root *first*: +the mount fails the run if the source path is missing. + +### Comparison mode: multistage + +`_gdpval_stages.cli_params` is the one place the stage shape is defined: + +```text +++multistage.enabled=true +++multistage.retry_inprocess=true +++multistage.stages='[{num_tasks: 45, partial_completion: {min_success_fraction: 0.9, + min_per_reference_success_fraction: 0.5, min_successful_rows_per_reference: 1, + waivable_failure_classes: [timeout_exceeded, transient], tolerate_unresolved: true}}, + {num_tasks: 220, num_models: 4}]' +++num_repeats=1 +``` + +Stage 1 samples **45 tasks against all references**; stage 2 judges the full **220** against +the **4 references nearest the stage-1 estimate**. Things to know before touching it: + ++ **`partial_completion` is not optional hardening.** Without it the orchestrator requires + *every* planned non-final rollout to carry usable battle evidence, so a single unresolved + rollout — one reaped job, one judge 400 — kills stage 2 after stage 1 has already been + paid for. That happened to real runs. `legitimate` failures stay non-waivable. ++ **The resume journal is fingerprinted on the stage shape**, so changing stages invalidates + in-flight runs. `num_repeats` is *not* fingerprinted — never flip it mid-campaign. ++ **There is no per-stage repeats key.** One inside a stage dict is silently ignored; + `++num_repeats` is rollout-collection level and applies to every stage. ++ **Never combine `++limit` with multistage.** It truncates the loaded dataset while stages + sample from the full distribution → 0 rollouts. The canonical config sets + `allow_limit_samples: false` so the launcher-level limiter cannot be believed either. ++ `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=3` bounds the in-process retry (`retry_inprocess`) at two + passes. The cap alone resolves nothing — `retry_inprocess` is what re-dispatches. + ## Judge -Rubric mode uses a single judge. **Comparison mode uses a 3-member panel** — -`openai/gpt-5.5`, `gcp/google/gemini-3.1-pro-preview`, -`aws/anthropic/bedrock-claude-opus-4-8` — one **sampled per trial**, all routed -through the single `gdpval_judge_model` proxy (`` from `.env`). -`++...judge_sampling_seed=42` makes that sampling reproducible. +Rubric mode uses a single judge. **Comparison mode uses a 3-member panel**, one +**sampled per trial**, all routed through the single `gdpval_judge_model` proxy +(`` from `.env`). `++...judge_sampling_seed=42` makes that sampling +reproducible, and `++...num_comparison_trials=4` sets the trials per rollout. + +**Each member carries its own generation overrides — copy them verbatim, they are part +of the methodology:** + +| Member | `model` | `create_params_overrides` | +|---|---|---| +| gpt-5.5 | `openai/openai/gpt-5.5` | `{reasoning_effort: medium}` | +| gemini-3.1-pro | `gcp/google/gemini-3.1-pro-preview` | `{extra_body: {reasoning_effort: high}}`, `handles_audio_video: true` | +| claude-opus-4.8 | `aws/anthropic/bedrock-claude-opus-4-8` | `{extra_body: {thinking: {type: enabled}}}` | + **Inject the key's VALUE, not its name:** `openai_api_key=$INFERENCE_API_KEY`. Passing an env-var *name* (e.g. via a `${...api_key}` interpolation that resolves to @@ -181,9 +294,10 @@ through the single `gdpval_judge_model` proxy (`` from `.en not a config error. One key covers all three panel members. + **Do not set `judge_responses_create_params_overrides.model`.** Pinning a model collapses the panel to a single judge, silently changing the scoring methodology. -+ **Throttles:** judge `max_concurrent_requests=10` and Stirrup `concurrency=220` are - the golden values — the judge rate-limits long before the served model does, so raise - these only after the judge logs are clean of 429s. ++ **Throttles:** judge `max_concurrent_requests=10`, Stirrup `concurrency=220` and + `preconvert_max_concurrent=30` are the canonical values — the judge rate-limits long + before the served model does, so raise these only after the judge logs are clean of + 429s. ## Preflight — what NEL validates, and what it does NOT @@ -226,7 +340,10 @@ mount source, and `raise ValueError` listing the missing ones **before** any | `GDPVAL_CONTAINER_PATH` | lit | SIF path — must equal the SIF bind-mount target | | `GDPVAL_REF_FILES_DIR` | lit:/gdpval_ref_files | shared-FS ref-file staging (node-local /tmp breaks multi-node Ray) | | `PERSIST_DELIVERABLES_DIR` | lit | where deliverables persist (see MLflow note) | -| `GDPVAL_MAX_TURNS` | lit (optional) | Stirrup turn cap (default 100; golden uses 250) | +| `GDPVAL_MAX_TURNS` | lit (optional) | Stirrup turn cap (default 100; canonical uses 250) | +| `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` | lit:3 | bounds Gym's in-process rollout retry at two passes | +| `JUDGE_API_KEY` | host (comparison) | upstream splits judge auth from `INFERENCE_API_KEY`; this template reuses the latter for both | +| `INTEGRATE_NVAPI_KEY` / `COMP_EVAL_NVCF_SERVICE_KEY` | host (comparison) | auxiliary Qwen3-235B / wildguard endpoints in the canonical config | | `NEL_INVOCATION_ID` | runtime | stable run id assigned by the validated launcher; do not use `SLURM_JOB_ID` | `INFERENCE_JUDGE_URL` is the judge host — config (from `.env`), substituted as the @@ -256,9 +373,10 @@ nemo_gym.gdpval`). ## num_repeats -**Use 1.** Both current goldens do, set with a top-level `++num_repeats=1` — it -works, and recent Gym pins already ship `num_repeats: 1` in -`benchmarks/gdpval/config.yaml`, so no `sed` patching is needed. +**Use 1.** The canonical config sets it with a top-level `++num_repeats=1` inside the +multistage block, and recent Gym pins already ship `num_repeats: 1` in +`benchmarks/gdpval/config.yaml`, so no `sed` patching is needed. It applies to every +stage — there is no per-stage repeats key (see "Comparison mode: multistage"). Historical only: pre-multistage single-reference configs used 2 (220 × 2 = 440 rollouts) and patched it with `sed` because the per-dataset key could not be set diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 8e009098d9f..17a0c7fcf18 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -52,6 +52,14 @@ prints the resolved spec, so check there if a run behaves unexpectedly. (SWE-bench), and `${MLFLOW_TRACKING_URI}` from `.env`. Run **`modelopttools:eval-config`** (Step 3b) to write them — it holds the canonical values, arch/region rules, and points to the per-benchmark `bench.yaml` source of truth. +- **`HARBOR_ECS_REGION` selects the sandbox region** (`sandbox.region` + + `cluster.container_env.AWS_DEFAULT_REGION`); upstream defaults it to `us-east-1` for + TB2.1 and `us-east-2` for SWE-bench. The two then treat the ECR differently: TB2.1's + repo **name tracks the region** + (`463701203462.dkr.ecr..amazonaws.com/harbor-`), while SWE-bench's is + pinned to `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench` whatever the + sandbox region is. So changing the region changes the TB2.1 ECR and not the SWE-bench + one — keep `${HARBOR_*_ECR_REPOSITORY}` in `.env` consistent with it. - `set -a && source .env && set +a` before running so `${VAR}` resolves. ## Architecture — where each piece runs @@ -154,9 +162,13 @@ with its own `run_id`, copying the shared `services:` block. catch. This pair is safe because it is what the TB2.1 + SWE-bench parity runs executed, not by construction. **Re-canary whenever either side moves**, and treat a post-allocation `ValidationError` mentioning an unknown/missing config field as skew, not a config bug. -- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch). Re-check - against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which is the - pin and does move. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. +- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch; + `gitlab-master.nvidia.com/dl/joc/competitive_evaluation/nemo-evaluator-next:0.5.0.1-harbor`). + Re-check against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which + is the single pin for TB2.1 + SWE-bench Verified + SWE-bench Multilingual and does move. + It also carries `nemo_evaluator_next_version`, which the canonical configs echo into the + MLflow tag of the same name — tag your runs with it too, so a score traces back to the + image that produced it. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). - **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600). A smaller proxy timeout silently truncates long agent turns. @@ -178,7 +190,16 @@ with its own `run_id`, copying the shared `services:` block. for the life of the run, full request + response bodies, and the whole list is re-serialized on each write. A long agentic run that is 400ing or rate-limiting (the failure this dumps diagnose) grows the proxy without bound — exactly the run you can least afford to lose. - Chain position is per benchmark (last for TB2.1, first for SWE-bench). + It is **first** in the chain for both TB2.1 and SWE-bench (it used to be last for TB2.1). +- **Interceptor lists replace wholesale on merge, they do not append.** A leaf that adds + one interceptor must restate the whole chain from the model fragment + (`http_pairs_dump` → `drop_params` → `consolidate_system` → `reasoning` → + `reasoning_replay`, plus SWE-bench's `system_message` + `turn_counter`). Half a chain is + a silent scoring change, not an error. +- **`proxy.model_traffic.capture_request_body: true` belongs on the service leaf** + (`services..proxy.model_traffic`), never in a shared/benchmark-level block: a + shared `services:` entry is alias-only with no `type`, which fails the service + discriminator in every other model's composed config. - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`. @@ -188,7 +209,9 @@ with its own `run_id`, copying the shared `services:` block. - **MLflow export — config + a post-run push.** Add `output.export: [mlflow]` + `export_config.mlflow` (hardcode `experiment_name: /` — `${USER}=root` in-container; tags `framework`/`model`/`temperature`/`top_p` + `checkpoint_path`/`benchmark` - for dashboard attribution). `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` + for dashboard attribution; canonical also sets `task_name: `, + `nemo-evaluator-next-version`, and `log_config_params: true`). + `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` Step 3b — canonical `mlflow.frontier-evals.nvidia.com`; **not** the `mlflow-nemo-evaluator` alias, whose 308 strips `/api/...` → 405). **SLURM does NOT auto-export** — push after the run with `nel-next.sh mlflow-push` (Run flow), which resolves the var and falls back diff --git a/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh b/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh index 28a19e534f6..f712127d9d9 100755 --- a/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh +++ b/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh @@ -25,8 +25,9 @@ # "$SKILL_DIR/scripts/gdpval-sif.sh" [] [--commit ] [--force|--check] # Persistent path on the target cluster's shared FS. # DEFAULTS to $GDPVAL_SIF_DIR (from .env) when omitted. A -# directory -> /$GDPVAL_SIF_NAME (default python-3.13.gdpval.sif, -# matching the example config); a *.sif path +# directory -> /$GDPVAL_SIF_NAME (default +# python-3.13.gdpval.gym-80e4fc.sif, matching the example config +# and the canonical GDPVAL_CONTAINER_PATH); a *.sif path # is used verbatim. Bind-mount this SAME dir into the eval # container at /gdpval/sif (see recipes/examples/gym/). # --commit NeMo Gym commit whose gdpval.def to build. Keep in sync @@ -51,8 +52,12 @@ set -euo pipefail # Keep GDPVAL_GYM_COMMIT in sync with install_on_the_fly.commit in the config. -GDPVAL_GYM_COMMIT="${GDPVAL_GYM_COMMIT:-dd41196f620f2af99947d776cbe5da9439d2a08d}" # pragma: allowlist secret -GDPVAL_SIF_NAME="${GDPVAL_SIF_NAME:-python-3.13.gdpval.sif}" +# df3e201d is the GDPval-AA v2 pin: its gdpval.def is the one the canonical +# gym-80e4fc image was built from (Debian trixie, python3.13=3.13.5 asserted at build +# time, the 419 published AA package pins, Office metric-substitute fonts, and no +# torch/keras/jax). Bump this and GDPVAL_SIF_NAME together. +GDPVAL_GYM_COMMIT="${GDPVAL_GYM_COMMIT:-df3e201d942f6397def1ec8c10037d29115fcbbd}" # pragma: allowlist secret +GDPVAL_SIF_NAME="${GDPVAL_SIF_NAME:-python-3.13.gdpval.gym-80e4fc.sif}" APPTAINER_BIN="${APPTAINER_BIN:-}" _log() { printf '\033[2m %s\033[0m\n' "$*" >&2; } @@ -128,11 +133,19 @@ fi [[ -n "$APPTAINER_BIN" ]] || _die "apptainer/singularity not found on PATH. Run on a node that has it \ (e.g. 'module load apptainer', or inside the eval image). This script does NOT copy a SIF from another cluster." -def_url="https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/${GDPVAL_GYM_COMMIT}/responses_api_agents/stirrup_agent/containers/gdpval.def" +def_base="https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/${GDPVAL_GYM_COMMIT}/responses_api_agents/stirrup_agent/containers" +# apptainer resolves %files SOURCES relative to the build's CWD, so the def cannot be +# built alone: stage it with its siblings and build from inside that dir. +build_dir="${sif_dir}/.gdpval-build.$$" tmp="${sif_dir}/.build.$$.${GDPVAL_SIF_NAME}" -def_local="${sif_dir}/.gdpval.$$.def" lock="${sif_dir}/.gdpval-sif.lock" +# _fetch — curl if available, else wget. +_fetch() { + if command -v curl >/dev/null 2>&1; then curl -fsSL "$1" -o "$2" + else wget -qO "$2" "$1"; fi +} + # --- build under a flock (double-checked) so concurrent runs don't double-build --- exec 9>"$lock" || _die "cannot open lock file: $lock" _log "acquiring build lock ($lock) ..." @@ -145,32 +158,53 @@ fi _log "building GDPVal SIF (this can take ~20-40 min)" _log " gym commit: ${GDPVAL_GYM_COMMIT}" -_log " def: ${def_url}" +_log " def: ${def_base}/gdpval.def" _log " dest: ${sif}" # Leave no temp artefacts if we are killed or exit early. $tmp is renamed on success, # so this only ever removes leftovers. -trap 'rm -f "$tmp" "$def_local"' EXIT -rm -f "$tmp" "$def_local" -# apptainer build cannot take a remote def URL as its source — fetch the def to a -# local file first, then build from it. -if command -v curl >/dev/null 2>&1; then curl -fsSL "$def_url" -o "$def_local" -else wget -qO "$def_local" "$def_url"; fi -[ -s "$def_local" ] || { rm -f "$def_local"; _die "failed to download def from $def_url"; } +trap 'rm -rf "$tmp" "$build_dir"' EXIT +rm -rf "$tmp" "$build_dir" +mkdir -p "$build_dir" || _die "cannot create build staging dir: $build_dir" + +# apptainer build cannot take a remote def URL as its source — fetch the def first. +_fetch "${def_base}/gdpval.def" "${build_dir}/gdpval.def" || true +[ -s "${build_dir}/gdpval.def" ] || _die "failed to download def from ${def_base}/gdpval.def" + +# The GDPval-AA v2 def declares %files (the published 419-package pin list, the apt +# closure, the arm64 exclusions, the sandbox verifier). Those sources resolve against +# the build CWD, so fetch every one named in the def — parsed rather than hardcoded, so +# a def that adds a file keeps working. +_files=$(awk ' + /^[[:space:]]*%files([[:space:]]|$)/ { inblk=1; next } + /^[[:space:]]*%/ { inblk=0 } + inblk && $1 !~ /^#/ && NF { print $1 } +' "${build_dir}/gdpval.def") +for _f in $_files; do + case "$_f" in + /*|*..*) _die "refusing to stage %files source outside the containers dir: $_f" ;; + esac + _log " staging %files source: $_f" + _fetch "${def_base}/${_f}" "${build_dir}/${_f}" || true + [ -s "${build_dir}/${_f}" ] || _die "def declares %files source '$_f' but it could not be fetched from ${def_base}/${_f}" +done + +# Absolute, because the build runs with CWD inside $build_dir. +_abs_tmp="$(cd "$(dirname "$tmp")" && pwd)/$(basename "$tmp")" # Prefer --fakeroot (needs an /etc/subuid entry for the build user); fall back to an # unprivileged build where fakeroot is unavailable. -if "$APPTAINER_BIN" build --fakeroot "$tmp" "$def_local"; then +if (cd "$build_dir" && "$APPTAINER_BIN" build --fakeroot "$_abs_tmp" gdpval.def); then : # A failed --fakeroot attempt can leave a partial $tmp behind, and apptainer refuses an # existing destination — clear it or the unprivileged fallback can never succeed. -elif rm -f "$tmp" && "$APPTAINER_BIN" build "$tmp" "$def_local"; then +elif rm -f "$_abs_tmp" && (cd "$build_dir" && "$APPTAINER_BIN" build "$_abs_tmp" gdpval.def); then _log "built without --fakeroot (unprivileged mode)" else - rm -f "$tmp" "$def_local" + rm -rf "$_abs_tmp" "$build_dir" _die "apptainer build failed (see output above)." fi -rm -f "$def_local" +rm -rf "$build_dir" # Atomic publish: a partial build never looks complete. -mv -f "$tmp" "$sif" || { rm -f "$tmp"; _die "failed to move built SIF into place: $sif"; } +mv -f "$_abs_tmp" "$sif" || { rm -f "$_abs_tmp"; _die "failed to move built SIF into place: $sif"; } _log "done: $sif" echo "$sif"