diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index da57288528f..4b574bfc53a 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -42,7 +42,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead: 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). -3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block. +3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block. 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r -c ` after (config-driven; see `references/nel-next.md`). Steps 1–9 below are currently validated with 0.2.6 — use them for everything else. @@ -65,18 +65,22 @@ tasks). If the user asks for GDPVal: (NVIDIA-internal: `modelopttools:eval-config` Step 3c); otherwise set `GDPVAL_SIF_DIR` in `.env` and build with `"$SKILL_DIR/scripts/gdpval-sif.sh"` (build-if-absent, no cross-cluster copy). Either way the mounted dir must contain - the file `GDPVAL_CONTAINER_PATH` names (template: `python-3.13.gdpval.sif`) — a + the file `GDPVAL_CONTAINER_PATH` names (canonical: + `python-3.13.gdpval.gym-80e4fc.sif`, the GDPval-AA v2 sandbox) — a name mismatch passes NEL's `test -d` check and the agent then silently runs - unsandboxed. Verify with `gdpval-sif.sh --check`. `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`, + unsandboxed. Verify with `gdpval-sif.sh --check`. The SIF, the Gym pin and the eval + image's Python move as **one unit** (reference: "Three things that move together"). + `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`, `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR`, and `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the config has a `pre_cmd`). Thinking mode is mandatory (non-thinking loses ~86%). 4. Run both dry-run and launch through `"$SKILL_DIR/scripts/nel-gdpval.sh"`; it enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale and avoids an unset `NEL_INVOCATION_ID` failure before client startup. - **`limit_samples` is inert on the gym path** (the gym runs all 220 tasks - regardless), so there is no cheap canary: watch the real run's first - ~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the - recipe's Canary section. + **`limit_samples` is inert on the GDPVal gym path** (the gym runs all 220 tasks + regardless; the config says so via `allow_limit_samples: false`), so there is no + cheap canary: watch the real run's first ~20–30 min for the SIF-sandbox line and + judge auth, and cancel if wrong. See the recipe's Canary section. (MRCR is the + exception — there `limit_samples` *does* reach the gym.) --- @@ -92,6 +96,8 @@ for an "AA" request. If the user asks for MRCR: 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it sets the context cap, dataset *and* metric prefix; the three are not comparable; set it in **both** `data_prep_params` and `collect_rollout_params`. + Upstream now carries 1M and 128K as two separate benchmarks with their own + manifests and metric keys — see the recipe. 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` + `transformers`; prepare fails without it) and diff --git a/plugins/modelopt/skills/evaluation/recipes/env.example b/plugins/modelopt/skills/evaluation/recipes/env.example index 2eb91e377f6..9a03c28fb58 100644 --- a/plugins/modelopt/skills/evaluation/recipes/env.example +++ b/plugins/modelopt/skills/evaluation/recipes/env.example @@ -40,6 +40,14 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # GDPVal (nemo_gym) also reuses INFERENCE_JUDGE_URL for its pairwise judge. # INFERENCE_JUDGE_URL=https:///v1 +# GDPVal COMPARISON mode only — upstream splits judge auth from INFERENCE_API_KEY and +# reaches two auxiliary endpoints. The rubric template in this skill reuses +# INFERENCE_API_KEY for the judge and needs none of these; the +# `modelopttools:eval-config` Step 3c conversion adds them. +# JUDGE_API_KEY= # 3-member judge panel auth +# INTEGRATE_NVAPI_KEY= # Qwen3-235B helper endpoint +# COMP_EVAL_NVCF_SERVICE_KEY= # allenai/wildguard safety server + # GDPVal (nemo_gym Stirrup agent) — agent web search. Secret; exported and read # by the harness. See recipes/tasks/gym/gdpval.md + references/gym-gdpval.md. # TAVILY_API_KEY= @@ -50,6 +58,13 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # /gdpval/sif. Convention: a per-user .cache dir. # GDPVAL_SIF_DIR=//.cache/gdpval/sif +# GDPVal (nemo_gym) — SIF filename inside GDPVAL_SIF_DIR. Only set this to point at a +# site-provided image whose name differs from the canonical one; it must equal the +# basename in the config's GDPVAL_CONTAINER_PATH, or NEL's `test -d` mount check passes +# and the agent silently runs code-exec unsandboxed. +# Default: python-3.13.gdpval.gym-80e4fc.sif (the GDPval-AA v2 sandbox). +# GDPVAL_SIF_NAME=python-3.13.gdpval.gym-80e4fc.sif + # GDPVal (nemo_gym) — Stirrup agent turn cap. Read at SUBMIT time from the # launching shell (the config uses ${oc.env:GDPVAL_MAX_TURNS,250}), so it must be # exported before `nel run`; setting it as a container env var has no effect. @@ -67,6 +82,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # AWS_ACCESS_KEY_ID= # AWS_SECRET_ACCESS_KEY= # NEL_NEXT_EVAL_IMAGE= # harbor eval image (arch-matched) — set by eval-config +# HARBOR_ECS_REGION= # Fargate sandbox region. Upstream defaults: us-east-1 +# # (Terminal-Bench), us-east-2 (SWE-bench). TB's ECR repo +# # NAME tracks this (harbor-); SWE-bench's does not. # HARBOR_ECR_REPOSITORY= # Terminal-Bench harbor ECR — set by eval-config -# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2) — set by eval-config +# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2, region-independent) — set by eval-config # MLFLOW_TRACKING_URI= # MLflow tracking URI (canonical frontier-evals host) — set by eval-config diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 9463c653062..1f7850f8782 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -45,17 +45,27 @@ services: proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below + model_traffic: {capture_request_body: true} # FEA-224 — set PER SERVICE, never in a shared block (an alias-only entry has no `type` and fails the service discriminator) + # Interceptor lists REPLACE wholesale on merge — if you add one, restate them all. interceptors: + # FEP-1104/1120 diagnostics — canonical position is FIRST (both TB2.1 and SWE-bench). + # Uncomment for a CANARY/debug run, drop it for the scored run: first_n caps only 200s, + # so every error pair (full req+res bodies) is retained in memory for the whole run and + # re-serialized on each write — unbounded growth exactly when the server errors. + # - name: http_pairs_dump + # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + # SWE-bench (OpenHands, multi-turn) inserts system_message + turn_counter HERE, before + # drop_params — don't lift that chain blind, see swebench_verified.md - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]} - # SWE-bench (OpenHands, multi-turn) adds turn_counter + system_message AND USES A DIFFERENT - # ORDER (drop_params before consolidate_system) — don't lift this chain — see swebench_verified.md - # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run: - # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for - # the whole run and re-serialized on each write — unbounded growth exactly when the server errors. - # - name: http_pairs_dump # canonical LAST in the chain (SWE-bench: first) - # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + - name: consolidate_system + # reasoning models only — normalize the reasoning field and replay it across turns. + # `reasoning_replay.mode` is per MODEL (think_tags default / native / both), not per + # benchmark. Drop both for instruct models. + # - name: reasoning + # - name: reasoning_replay + # config: {mode: think_tags} node_pool: gpu benchmarks: @@ -67,10 +77,12 @@ benchmarks: timeout_strategy: max # canonical TB2.1; "task" = leaderboard-comparable agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-1 # MUST match the region in ${HARBOR_ECR_REPOSITORY} (SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}) + # TB2.1's ECR repo NAME tracks the region (harbor-); SWE-bench's is pinned to + # us-west-2/harbor-swebench whatever the sandbox region is (us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}). + region: ${HARBOR_ECS_REGION:-us-east-1} ecr_repository: ${HARBOR_ECR_REPOSITORY} # from modelopttools:eval-config concurrency: 50 - log_stream_prefix: terminalbench21-??? + log_stream_prefix: terminalbench-21-??? # canonical stem + - cluster: type: slurm @@ -87,7 +99,7 @@ cluster: HF_HOME: /cache/huggingface AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} - AWS_DEFAULT_REGION: us-east-1 # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} + AWS_DEFAULT_REGION: ${HARBOR_ECS_REGION:-us-east-1} # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} LLM_API_KEY: "no-key-needed" mount_home: false auto_resume: true @@ -110,4 +122,12 @@ output: exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. - tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} + tags: + framework: vllm + model: "???" + checkpoint_path: "???" + benchmark: "???" + task_name: "terminal-bench-2.1" # upstream benchmark slug (SWE-bench: swebench-verified) + nemo-evaluator-next-version: "0.5.0.1" # eval_image version, so a score traces to its image + temperature: '1.0' + top_p: '0.95' diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml index 6e5eef250fa..91e39c144fb 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml @@ -14,7 +14,7 @@ # limitations under the License. # # ============================================================================= -# GDPVal (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode. +# GDPVal-AA v2 (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode. # Self-deploys a checkpoint via vLLM on one SLURM node. One gym eval per config. # # Read recipes/tasks/gym/gdpval.md + references/gym-gdpval.md first — they @@ -42,20 +42,25 @@ defaults: # reference deliverables and runs on the public gym image, so it works standalone. # rubric — standalone LLM-judge scoring, 0-1 reward per deliverable. (this template) # comparison — pairwise vs anchored reference deliverables; the only mode that yields -# an AA-comparable ELO / win-rate. It needs a reference set, a newer gym -# image, and its own overrides — do NOT just flip reward_mode here. +# an AA-comparable ELO, and the mode the upstream GDPVal-AA-V2 benchmark +# is defined in. It needs the 12-reference set, a py3.13 gym image, a +# 3-judge panel and multistage overrides — do NOT just flip reward_mode. # To run comparison, NVIDIA-internal users should follow `modelopttools:eval-config` # Step 3c, which converts this config (container override + reference_models map + # mounts + multistage). See references/gym-gdpval.md "Scoring modes". gdpval: reward_mode: rubric -# GDPVal pairwise judge. base_url is config (from .env), not a secret, so no -# export needed; only api_key (INFERENCE_API_KEY) is exported and read by the -# harness. Keep the judge fixed across comparable runs. +# GDPVal judge. base_url is config (from .env), not a secret, so no export needed; +# only api_key (INFERENCE_API_KEY) is exported and read by the harness. Keep the judge +# fixed across comparable runs. +# RUBRIC mode (this template) scores with this single judge. COMPARISON mode routes a +# 3-member panel — gpt-5.5 / gemini-3.1-pro / claude-opus-4.8, one SAMPLED PER TRIAL at +# judge_sampling_seed=42 — through this same proxy; the panel is added by the +# eval-config Step 3c conversion, not here. See references/gym-gdpval.md "Judge". gdpval_judge: base_url: # from .env (/v1 base); shared inference host - model: gcp/google/gemini-3.1-pro-preview # Gemini 3.1 Pro; use an equivalent on your endpoint if needed + model: gcp/google/gemini-3.1-pro-preview # rubric judge; also the panel's default member api_key: INFERENCE_API_KEY # doc only; common_params injects the VALUE ($INFERENCE_API_KEY). # An env-var NAME reaches the proxy as a literal -> opaque judge 500. @@ -136,7 +141,13 @@ evaluation: TAVILY_API_KEY: host:TAVILY_API_KEY # Stirrup agent web search UV_CACHE_DIR: lit:/cache/uv # Apptainer SIF for the Stirrup per-task code-exec sandbox (must match the mount above). - GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.sif + # gym-80e4fc is the GDPval-AA v2 sandbox: Debian trixie, python 3.13.5 asserted at + # build time, the 419 published AA package pins, and the Office metric-substitute + # fonts earlier images lacked. It is versioned WITH the Gym pin below — move both or + # neither ($SKILL_DIR/scripts/gdpval-sif.sh defaults to this pair). + GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif + # Bounds Gym's in-process rollout retry. Canonical is 3 (= two retry passes). + NEMO_GYM_MAX_ROLLOUT_ATTEMPTS: lit:3 # Shared-FS staging for ref files; node-local /tmp breaks multi-node Ray. GDPVAL_REF_FILES_DIR: lit:/gdpval_ref_files # Deliverables land under /results (already mounted) so they persist. The @@ -205,12 +216,25 @@ evaluation: # as a non-git directory, so `install_on_the_fly.commit` below is silently # ignored here (the prepare step logs "/opt/Gym is not a git repo; using baked-in # Gym version") — the pin only applies on images where /opt/Gym IS a git repo. + # ON A GIT-BACKED IMAGE THE PIN APPLIES, AND IT NEEDS PYTHON >= 3.13.14. The + # canonical config pairs it with a py3.13 gym runtime image plus + # `runtime_python: /usr/local/bin/python3.13` (uncomment below). A py3.12 image + # + this pin fails at venv creation, not at scoring. container: nvcr.io/nvidia/eval-factory/nemo-gym:26.05 # pin a verified tag nemo_evaluator_config: config: params: extra: nemo_gym: + # limit_samples is INERT on the gym path; canonical says so explicitly + # so a "2-sample smoke test" can't be believed. Also required by + # multistage (comparison mode): ++limit truncates the loaded dataset + # while stages sample from the full distribution -> 0 rollouts. + allow_limit_samples: false + # Uncomment WITH a py3.13 gym runtime image above — the pinned Gym + # declares requires-python >= 3.13.14. Move image + interpreter + pin + # as one unit. + # runtime_python: /usr/local/bin/python3.13 install_on_the_fly: url: https://github.com/NVIDIA-NeMo/Gym # Gym version. BUMPING THIS REQUIRES REBUILDING THE SIF from the matching @@ -218,15 +242,17 @@ evaluation: # old SIF + new gym silently degrades deliverables. Rebuild with # `gdpval-sif.sh --commit ` to a new name + repoint GDPVAL_CONTAINER_PATH. # See references/gym-gdpval.md "Rebuild the SIF when the Gym version changes". - # Current golden pin (updated GDPVal task-sampling algo, which drives - # multistage stage-1 selection). Its gdpval.def is byte-identical to - # 049b1fd0…, so a python-3.13 SIF built from either commit is valid — - # no rebuild when moving between them. + # Current canonical pin (GDPval-AA v2). Carries the AA reference-ELO + # anchor refresh, the three multistage reference anchors, long-context + # budgeting (completion floor 8192), and the gdpval.def the gym-80e4fc + # SIF above was built from. Runs on it are NOT strictly comparable to + # runs on the older dd41196f pin — the agent's prompt and generation + # behaviour both changed. # NOTE: this pin is INERT on the public nemo-gym image (see the # `container:` comment above) — it only takes effect on an image where # /opt/Gym is a git repo. Confirm via "=== NeMo Gym commit ===" + SHA # in the client log before crediting it with any behaviour change. - commit: dd41196f620f2af99947d776cbe5da9439d2a08d # pragma: allowlist secret + commit: df3e201d942f6397def1ec8c10037d29115fcbbd # pragma: allowlist secret command: | set -ex cd /opt/Gym @@ -327,6 +353,9 @@ evaluation: # No judge_responses_create_params_overrides.model here: the judge model is # already set by openai_model=${gdpval_judge.model}, and pinning it a second # way silently collapses comparison mode's 3-member judge panel to one judge. + # Comparison mode additionally appends num_comparison_trials=4, + # judge_sampling_seed=42, the judge_panel list, the reference_models map and + # the multistage block — all from eval-config Step 3c, none of them here. # num_repeats=1 — both current goldens. No reference_* overrides here: # the single-reference keys conflict with comparison mode's reference_models # map (added by modelopttools:eval-config Step 3c). @@ -357,7 +386,7 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | GDPVal rubric | T=1.0, top_p=0.95, num_repeats=1' + description: 'CHANGEME-served-model-name | GDPVal-AA v2 rubric | T=1.0, top_p=0.95, num_repeats=1' log_logs: true only_required: false tags: diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 01c1c9a42c4..1b2d4fe8267 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -86,9 +86,11 @@ deployment: gpu_memory_utilization: 0.95 # golden raises this from 0.85 to fit the 1M KV cache # Golden serves in REASONING mode (thinking forced on via chat_template_kwargs # below), so set the model's --reasoning-parser. - # After sizing num_nodes / num_instances / TP / DP above, append - # `--max-num-seqs N` where N = ceil(parallelism / num_instances / DP) — the GYM - # rule, not the generic SKILL one. Golden: ceil(256/4/2) = 32. + # After sizing num_nodes / num_instances / TP / DP above, append `--max-num-seqs N`. + # Do NOT derive N from `parallelism`: the canonical bench.yaml now sets parallelism + # 512 DELIBERATELY ABOVE total server capacity so 1M-token requests queue at the + # endpoint instead of leaving replicas idle. Size N from what a replica's KV can hold + # (the golden ran 32 per replica × 8 replicas = 256 in flight against parallelism 512). # Treat that as a CEILING, not a target. Like AA-LCR (recipes/tasks/aa/lcr.md) # this task is KV-bound, and far more so: ~1M input tokens per request vs LCR's # ~120K. Too high and vLLM preempts, and recomputing a 1M-token prefill makes the @@ -131,10 +133,16 @@ evaluation: params: temperature: 1.0 top_p: 0.95 - # Gym client concurrency, TOTAL across instances — not a per-server cap. - parallelism: 256 + # Gym client concurrency, TOTAL across instances — not a per-server cap, and + # intentionally ABOVE it: the canonical bench.yaml uses 512 so the endpoint + # always has queued work while Gym's own dispatch stays bounded. Drives both + # ++num_samples_in_parallel and ++global_aiohttp_connector_limit_per_host. + parallelism: 512 request_timeout: 36000 # 1M-token prefills are slow max_retries: 10 + # null = full run. Set N for a canary: unlike GDPVal, MRCR forwards it to the + # gym as ++limit (see the gated line in collect_rollout_params below). + limit_samples: # Uncapped: answers reproduce a full earlier turn, and any cap truncates # it and craters SequenceMatcher.ratio (see max_output_tokens=null below). max_new_tokens: @@ -292,9 +300,12 @@ evaluation: ++reuse_existing_data_preparation=true ++skip_venv_if_present=true ++upload_rollouts_to_wandb=false + ++observability_enabled=true + ++model_call_capture_dir={{config.output_dir}}/model_calls ++responses_create_params.temperature={{config.params.temperature}} ++responses_create_params.top_p={{config.params.top_p}} ++responses_create_params.max_output_tokens=null + {% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %} common_params: >- ++num_repeats=1 ++policy_base_url={{target.api_endpoint.url}} diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index bbd70551cb9..289cf7c672d 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -8,6 +8,11 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. > **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in > nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image > pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run. +> +> That dir also ships `bench_direct.yaml`, a **different backend**: the Gym/opencode +> route (`++num_samples_in_parallel=1024`, its own Gym commit pin, opensandbox) used by +> the P/D recipes. `benchmark_backend: evaluator` — the harbor path described here — is +> what this skill runs. Don't mix values between the two. ## Task-specific values (canonical `bench.yaml`) @@ -16,12 +21,15 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | scope | 500 Python tasks × `repeats: 5` | -| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | +| `max_concurrent` / `sandbox.concurrency` | `15` (canonical `bench.yaml`); the reviewed per-model leaves keep it. A separate customer-launch track (`reviewed_configs/clu/…`) runs `repeats: 3` / `concurrency: 10` — **not** AA-comparable, don't borrow its numbers | | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | -| `sandbox.region` | `us-east-2` | -| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-2}` (differs from TB2.1's `us-east-1` default) | +| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` → `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench`. **Hardcoded us-west-2 upstream** — unlike TB2.1 the repo does *not* track `HARBOR_ECS_REGION`; moving the sandbox region leaves the ECR alone | | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) *(shared — see `references/nel-next.md`)* | -| `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | +| `cluster.container_env.AWS_DEFAULT_REGION` | `${HARBOR_ECS_REGION:-us-east-2}` (match `sandbox.region`) | +| `sandbox.log_stream_prefix` | canonical stem `swebench---` | +| `output.export_config.mlflow.tags` | canonical adds `task_name: swebench-verified` and `nemo-evaluator-next-version: 0.5.0.1` | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block | | `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) | | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | @@ -32,7 +40,7 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. benchmarks: - playbook: swebench_verified repeats: 5 - max_concurrent: 15 # keep == sandbox.concurrency; per-model configs may raise both + max_concurrent: 15 # canonical; keep == sandbox.concurrency instruction_template: /configs/prompts/swebench_instruction.md # mounted (see gotcha) solver: service: @@ -40,7 +48,7 @@ benchmarks: run_timeout: 10800 agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-2 + region: ${HARBOR_ECS_REGION:-us-east-2} # canonical; the ECR below stays us-west-2 ecr_repository: ${HARBOR_SWEBENCH_ECR_REPOSITORY} concurrency: 15 log_stream_prefix: swebench-verified-- @@ -88,10 +96,16 @@ OpenHands runs ~200 turns/task. The canonical config adds a `system_message` interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) plus `turn_counter`. -**Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` -comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it -retains every error pair in memory for the whole run (`references/nel-next.md`); drop it -from the scored config. +**Order.** `http_pairs_dump` is **first**, then `system_message` → `turn_counter` → +`drop_params` → `consolidate_system` → `reasoning` → `reasoning_replay`. (TB2.1 now puts +`http_pairs_dump` first too; the SWE-bench-only part is the `system_message` + +`turn_counter` pair.) `http_pairs_dump` is canary/diagnostic-only — it retains every error +pair in memory for the whole run (`references/nel-next.md`); drop it from the scored config. + +**Interceptor lists replace wholesale on merge.** The reviewed leaf restates the model +fragment's whole chain to add two entries — it does not append. Adding `system_message` +without re-listing `drop_params`/`consolidate_system`/`reasoning`/`reasoning_replay` +silently drops them. ```yaml proxy: @@ -127,11 +141,21 @@ Fargate quota and `N × gpus_per_node` against your allocation. ## Score Extraction -Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): +Report **`pass@1`** — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): the resolved rate over the 500 tasks, **already averaged over repeats** (nel-next reports a single `pass@1`; there is **no `avg-of-N` key** like the 0.2.6 nemo-skills -metrics). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer -table) in the run dir or `nel eval report -r `, then push to MLflow with +metrics). MLflow logs it as `pass_at_1`. + +> The upstream `manifest.yaml` lists its metric key as **`mean/reward`**, not `pass@1` — +> that is the key the Gym/direct (`bench_direct.yaml`) recipes emit, and it is shared by +> the whole SWE-bench family (Verified / Multilingual / Pro). On the harbor path the +> certificates in `reviewed_configs/…/swebench_verified/certificates/` record `pass@1` +> (e.g. `0.516`), so `pass@1` is still what you report for a nel-next run. Quote the key +> name alongside the number so a `mean/reward` from the direct path is never compared +> against it. + +Read from `report.md` (Benchmark / Scorer table) in the run dir or +`nel eval report -r `, then push to MLflow with `nel-next.sh mlflow-push -r -c ` (SLURM doesn't auto-export). Keep `timeout_strategy` + the instruction/system prompt fixed across baseline vs quantized for a valid delta. diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index 42a5366735a..b022d74c498 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -18,23 +18,35 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | agent | `terminus-2` (from the playbook) | | `repeats` | `8` (AA / leaderboard count — don't lower for a scored run) | | sandbox | `ecs_fargate`, `stateful: true` (agent + verifier share one container) | -| `sandbox.region` | match the region in `${HARBOR_ECR_REPOSITORY}` (us-east-1 with the eval-config default) | -| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by the `modelopttools:eval-config` skill (internal harbor ECR) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-1}` — one env var now drives region **and** repo name | +| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by `modelopttools:eval-config`; canonical value is `463701203462.dkr.ecr..amazonaws.com/harbor-`, i.e. the repo **name tracks the region** (`harbor-us-east-1`). Changing region means changing both halves. | | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | +| `sandbox.log_stream_prefix` | canonical stem `terminalbench-21`; the reviewed leaf appends model + framework (`terminalbench-21-nano-v35-vllm`) | | `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) | | timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable | | `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) *(shared — see `references/nel-next.md`)* | | `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` *(shared — see `references/nel-next.md`)* | -| `http_pairs_dump` | **last** in the interceptor chain — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `http_pairs_dump` | **first** in the interceptor chain (same position as SWE-bench) — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block, which would inject an alias-only entry with no `type` into other models' composed configs | +| `output.export_config.mlflow.tags` | canonical adds `task_name: terminal-bench-2.1` and `nemo-evaluator-next-version: 0.5.0.1` alongside `model`/`framework` | | scope | 89 tasks × `repeats: 8` | These values mirror the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in `configs/shared/nel_next_containers.yaml`. See `references/nel-next.md` + the eval-config -"source of truth" note. The `benchmarks:` block (drop into the example template): +"source of truth" note. + +> **`bench_direct.yaml` is a different backend, not an alternative spelling.** The same +> upstream dir now also ships a Gym-native route (Terminus 2 agent + opensandbox, +> `++num_samples_in_parallel=512`, its own Gym commit pin) used by the P/D recipes. It +> scores the same 2.1 task set but through different machinery; `benchmark_backend: +> evaluator` — the harbor path below — is what this skill runs and what the nel-next +> deltas here describe. Don't mix values between the two. + +The `benchmarks:` block (drop into the example template): ```yaml benchmarks: @@ -49,10 +61,10 @@ benchmarks: llm_kwargs: timeout: 3600 # per-request LLM timeout (canonical) sandbox: - region: us-east-1 # must match the region in ${HARBOR_ECR_REPOSITORY} + region: ${HARBOR_ECS_REGION:-us-east-1} # canonical; repo name below must track it ecr_repository: ${HARBOR_ECR_REPOSITORY} # from eval-config (internal harbor account/region) concurrency: 50 - log_stream_prefix: terminalbench21-- + log_stream_prefix: terminalbench-21-- ``` `cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor`) and the AWS creds @@ -66,7 +78,8 @@ it is purely a wall-clock lever. `shards: 4` suits 89 × r8 = 712 trials. Check ## Score Extraction -Report **`pass@1`** only — benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): +Report **`pass@1`** — the key the upstream `manifest.yaml` certifies on, and the only +one to quote. Benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): the resolved rate over the 2.1 task set, **already averaged over repeats** (a single `pass@1`; no `avg-of-N` key). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer table) or `nel eval report -r `, then push to MLflow with diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md index c8f581995a1..2208aaf7141 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/gdpval.md @@ -1,16 +1,26 @@ -# GDPVal (NeMo Gym "Stirrup" agent) +# GDPVal-AA v2 (NeMo Gym "Stirrup" agent) ## Task Details - Reference: `references/gym-gdpval.md` (SIF build, gym machinery, deploy sizing, - scoring modes, failure modes) — **read it before editing a GDPVal config.** + scoring modes, reference ladder, multistage, failure modes) — **read it before + editing a GDPVal config.** +- Upstream config: `configs/benchmarks/gdpval-aa-v2/` → `configs/benchmarks/gym/gdpval/` + (`bench.yaml` + `refs.yaml`) in nvidia-eval-factory-benchmarking + (`dl/JoC/competitive_evaluation/…`) — the source of truth; re-read it before a scored run. - Upstream README: GDPVal is an **agentic** benchmark: the Stirrup agent produces office/PDF -deliverables inside a per-task Apptainer code-exec sandbox, then a pairwise/rubric -judge (**Gemini 3.1 Pro**) scores them. It is the most resource-intensive benchmark -in the suite — **220 tasks**, `num_repeats=1`, 4 judge trials per rollout. +deliverables inside a per-task Apptainer code-exec sandbox, then a judge scores +them — one rubric judge, or in comparison mode a **3-member panel** (gpt-5.5 / +gemini-3.1-pro / claude-opus-4.8) sampled per trial. It is the most +resource-intensive benchmark in the suite — **220 tasks**, `num_repeats=1`, 4 +comparison trials per rollout. + +**Upstream is now GDPVal-AA v2** — a rebuilt sandbox matching the published AA +environment, a refreshed 12-reference ELO ladder, and a two-stage anchored fit. A v2 +score is not comparable to a v1 one. It is currently validated with the **0.2.6 `nel` launcher** as a `nemo_gym` task (NOT nel-next), so Steps 1–9 apply — but with the branch differences below. @@ -33,14 +43,19 @@ suite membership, so read that per-task, not from the path. - **Apptainer SIF sandbox** — prefer a site-provided SIF; otherwise `$SKILL_DIR/scripts/gdpval-sif.sh` builds one into `$GDPVAL_SIF_DIR` (build-if-absent, never copied between clusters). Missing/misnamed → **silent** unsandboxed exec. + Canonical is `python-3.13.gdpval.gym-80e4fc.sif`, and it is versioned **with** the Gym + pin and the eval image's Python — see the reference's "Three things that move together". - **Thinking mode is mandatory** — non-thinking loses ~86% of pairwise judgements. Serve with the model's `--reasoning-parser` and force it on via the adapter's `chat_template_kwargs`. - **Scoring:** `rubric` (template default, no references, no ELO) vs `comparison` - (the AA-comparable `normalized_elo`; a conversion, not a flag flip). + (the AA-comparable `normalized_elo`; a conversion, not a flag flip). Comparison adds + the 12-reference ladder and a **two-stage** fit: stage 1 samples 45 tasks against all + references, stage 2 judges all 220 against the 4 nearest the stage-1 estimate. - Needs `INFERENCE_API_KEY`, `TAVILY_API_KEY`, `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR` in `.env`, plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the config has a - `pre_cmd`). + `pre_cmd`). Comparison mode additionally wants `JUDGE_API_KEY` and the auxiliary + endpoint keys — see the reference's "Env vars". All of the above — SIF handling, the SIF↔Gym-commit coupling, scoring modes, judge panel, preflight and failure modes — is detailed in **`references/gym-gdpval.md`**. @@ -64,7 +79,12 @@ current goldens use it. A full 220-task run of a large MoE typically needs multi **`++…params.limit_samples=N` is inert on the gym path.** The gym does its own data prep and rollout collection, so the launcher-level limiter is ignored: you get the full 220-task run. Do not use it believing you launched a two-task smoke test — this -is the heaviest benchmark in the suite. +is the heaviest benchmark in the suite. The canonical config states this in the config +itself (`extra.nemo_gym.allow_limit_samples: false`), and the template carries it too. + +**`++limit` is worse than useless in comparison mode**: it truncates the loaded dataset +while the stages sample from the full distribution, so you get **0 rollouts**. (MRCR is +the opposite — there `limit_samples` *does* reach the gym. Don't carry the habit across.) There is no cheap sample-limited canary. Instead, **launch the real run and treat its first ~20–30 minutes as the canary**, cancelling if any of these is wrong: @@ -77,8 +97,12 @@ grep -ciE " 401 | 403 |Internal Server Error" $RD/artifacts/nemo_gym_logs/gdpval wc -l $RD/artifacts/evaluator_rollouts.jsonl # rollouts flowing ``` -In comparison mode stage 1 (45 tasks) is a natural early checkpoint — an ELO estimate -appears before the full 220-task stage 2 starts. +In comparison mode stage 1 (45 tasks, all references) is a natural early checkpoint — an +ELO estimate appears before the full 220-task stage 2 starts. Stage 1 runs with +`partial_completion` gates (≥0.9 overall success, ≥0.5 per reference, ≥1 judged row per +reference; `timeout_exceeded` and `transient` waivable) so one reaped rollout or one judge +400 no longer forfeits stage 2 — if you see stage 2 refuse to start, check those gates +before blaming the model. ## Score Extraction @@ -88,29 +112,37 @@ appears before the full 220-task stage 2 starts. **The reported GDPVal score is `normalized_elo`** — the AA 0–1 scale, comparable across models and to the published AA index. `eval_elo` is the same fit on the raw -Elo axis (`normalized_elo = (eval_elo - 500) / 2000`); quote it as supporting -detail, not as the score. +Elo axis (`normalized_elo = (eval_elo - 500) / 2000`, fitted across all models); quote it +as supporting detail, not as the score. + +**Read it from the FINAL STAGE key, not the flat one.** The upstream manifest certifies +on `comparison/stage_1/normalized_elo` (stage index 1 = the 220-task stage), because the +flat `comparison/normalized_elo` is emitted **only when that stage is exactly complete** — +a few lost rollouts blank it out while the stage-scoped key still carries the fit. A +missing flat key is therefore not evidence the run failed to score. The final numbers live in **`artifacts/results.yml`** (authoritative, local) and are mirrored to MLflow. Read them by metric name: | Mode | Metric (results.yml → `groups.nemo_gym.metrics..scores..value`) | | --- | --- | -| comparison | `gdpval_stirrup_agent/comparison/normalized_elo` ← **REPORT THIS** (AA 0–1 scale) | -| comparison | `gdpval_stirrup_agent/comparison/eval_elo` (raw Elo; supporting detail) | +| comparison | `gdpval_stirrup_agent/comparison/stage_1/normalized_elo` ← **REPORT THIS** (AA 0–1 scale, final stage) | +| comparison | `gdpval_stirrup_agent/comparison/stage_1/eval_elo` (raw Elo; supporting detail) | +| comparison | `gdpval_stirrup_agent/comparison/stage_0/eval_elo` — the stage-1 (45-task) estimate; an early checkpoint, **not** the score | +| comparison | flat `gdpval_stirrup_agent/comparison/{normalized_elo,eval_elo}` — same quantity, but present only on an exactly-complete final stage | | comparison | `gdpval_stirrup_agent/comparison/win_rate`, `/judged`, `/wins`, `/losses`, `/ties` | | comparison | per-reference: `gdpval_stirrup_agent/comparison/ref//{win_rate,wins,losses,ties,judged}` | -| comparison | per-stage estimate: `gdpval_stirrup_agent/comparison/stage_0/eval_elo` (stage 1, all refs) — the **final** value is the top-level one, from the last stage | | rubric | mean of `reward` across `artifacts/evaluator_rollouts.jsonl` (per-rollout 0–1) | ```bash -# COMPARISON mode — final score from the local results file (no MLflow needed) +# COMPARISON mode — final score from the local results file (no MLflow needed). +# stage_1 first (what upstream certifies on); the flat key is the fallback. python3 -c " import yaml m=yaml.safe_load(open('//nemo_gym.0/artifacts/results.yml'))['groups']['nemo_gym']['metrics'] -for k in ('normalized_elo','eval_elo','win_rate'): +for k in ('stage_1/normalized_elo','stage_1/eval_elo','normalized_elo','eval_elo','win_rate'): n=f'gdpval_stirrup_agent/comparison/{k}' - print(k, '=', m[n]['scores'][n]['value'])" + if n in m: print(k, '=', m[n]['scores'][n]['value'])" # RUBRIC mode (the template default) — there is no ELO; average the per-rollout reward python3 -c " @@ -126,8 +158,9 @@ comparison run logs **~200 metrics and most of them are per-reference**, so the headline is easy to miss: ```text -nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/normalized_elo <- report this -nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/eval_elo +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/stage_1/normalized_elo <- report this +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/stage_1/eval_elo +nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/normalized_elo <- flat fallback nemo_gym_gdpval_stirrup_agent/key_metrics/comparison/win_rate ``` diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md index 2123d1c31c1..fd8a3f08040 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md @@ -5,6 +5,9 @@ - Benchmark: - Resource server: - Dataset: `openai/mrcr` (HF, gated → `HF_TOKEN`) +- **Source of truth:** `configs/benchmarks/mrcr/bench.yaml` + `manifest.yaml` (1M) and + `configs/benchmarks/mrcr-128k/` (128K) in nvidia-eval-factory-benchmarking + (`dl/JoC/competitive_evaluation/…`) — re-check before a scored run. Long-context retrieval. Each task is a long multi-turn conversation with N near-identical "needle" responses; the model must reproduce the Nth verbatim @@ -60,6 +63,21 @@ The n3 variants drop over-long samples, so all three are different datasets and candidate, and set it in **both** `data_prep_params` and `collect_rollout_params` — changing one prepares one dataset and rolls out another. +The metric prefix column is the **manifest metric key**: upstream certifies 1M on +`mrcr_n3_1m_benchmark_simple_agent/pass@1/accuracy` and 128K on +`mrcr_n3_128k_benchmark_simple_agent/pass@1/accuracy`. + +**128K is now its own upstream benchmark, on a newer launcher schema.** +`configs/benchmarks/mrcr-128k/` is a separate slug with its own manifest, and its +`bench.yaml` uses the condensed gym block — `benchmark: mrcr/config_n3_128k`, +`runtime_dependencies: tiktoken transformers`, `prepare_args` / `run_args` — against the +`gym_runtime` container, instead of this template's `data_prep_params` / +`collect_rollout_params` / inline `command:` bootstrap. Both reach the same Gym pin +(`a431501a`); the condensed form just moves the bootstrap into the runtime image. This +template stays on the explicit form, which is what the 1M `bench.yaml` still ships. +If you switch a config to the condensed schema, switch the container with it — the +`pre_cmd` and `command:` blocks here have no effect there. + The `num_repeats` column is what each variant **declares upstream**, not necessarily what runs: for `type: benchmark` datasets the value is a placeholder and the runner decides. The template therefore pins `++num_repeats=1` in @@ -89,10 +107,16 @@ deliberately. **Do not change repeat counts when aligning to a golden.** - Fan out via `execution.num_nodes` / `num_instances` (HAProxy pattern A — `references/multi-node.md`). **Size these from the cluster's GPUs-per-node**, do not copy: pick TP for the model, fill the node with DP, then choose instances for - the replica count you want. `parallelism` is the total across instances, so - `--max-num-seqs = ceil(parallelism / num_instances / DP)`. The golden ran 4 nodes - × (TP2 × DP2) on 4-GPU nodes = 8 replicas, `ceil(256/4/2) = 32` each; the same 8 - replicas on 8-GPU nodes is 2 × (TP2 × DP4). + the replica count you want. The golden ran 4 nodes × (TP2 × DP2) on 4-GPU nodes = + 8 replicas; the same 8 replicas on 8-GPU nodes is 2 × (TP2 × DP4). +- **`parallelism` is 512 and is NOT a server cap — do not derive `--max-num-seqs` + from it.** The canonical `bench.yaml` raised it from 256 with the comment that MRCR + prompts run to 1M tokens, "so allow the endpoint to queue requests while keeping + Gym's client concurrency bounded". It sets `++num_samples_in_parallel` and + `++global_aiohttp_connector_limit_per_host`; the server's in-flight cap is + independent and deliberately smaller (the golden's 8 replicas × `--max-num-seqs 32` + = 256, half of 512 — the rest queue). The old `ceil(parallelism / num_instances / DP)` + identity was a coincidence at 256 and is wrong at 512. - **`--max-num-seqs` is a ceiling, not a target.** MRCR is the most KV-bound task in the skill — ~1M input tokens per request against AA-LCR's ~120K — so AA-LCR's rule applies harder: oversubscribe and vLLM preempts, and recomputing a 1M-token @@ -139,28 +163,28 @@ NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image. ## Canary -MRCR's gym path takes `++limit=N` (the launcher-level `limit_samples` does not -reach the gym). **Not verified on this pinned commit** — treat the first ~30 min -of the real run as the canary, as the GDPVal recipe does. `++limit` caps rollouts -only: the 1M tokenize/drop-over-long **prepare pass still runs in full**, so a -5-sample canary is not cheap. Append it to `collect_rollout_params` in your copy -of the YAML — easier than re-pasting the whole folded scalar through `-o`: - -```yaml - collect_rollout_params: >- - ... - ++limit=5 # canary only — remove for the scored run -``` +**`limit_samples` now reaches the gym.** The canonical `bench.yaml` gates the override +on it — `{% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %}` +— and the template carries the same line, so `-o ++config.params.limit_samples=5` (or +editing `limit_samples:` in your copy) produces `++limit=5`. This is the opposite of +GDPVal, where the launcher-level limiter is inert. -Then watch the first ~30 min of the real run: +`++limit` caps **rollouts only**: the 1M tokenize/drop-over-long **prepare pass still +runs in full**, so a 5-sample canary is not cheap, and `pass@1` from a limited run is +not a score. Also watch the first ~30 min of the real run: ```bash RD=//nemo_gym.0 grep -A1 "=== NeMo Gym commit ===" $RD/logs/client-*.log | grep -c a431501a # pin applied grep -ciE "ModuleNotFoundError|tiktoken" $RD/logs/client-*.log # pre_cmd didn't take wc -l $RD/artifacts/evaluator_rollouts.jsonl # rollouts flowing +ls $RD/artifacts/model_calls | head # observability capture on ``` +`++observability_enabled=true` + `++model_call_capture_dir` are canonical and write +per-call records under `artifacts/model_calls` — useful for diagnosing the prefix-gate +failure below. They grow with the run; drop them only if you are short on space. + **Preempted vs timed out.** A 1M run routinely exceeds 4h. `TIMEOUT` auto-resumes from the response cache; `CANCELLED by ` (preemption) does not — its chained job exits in ~20s (`…finished with 'CANCELLED…' state. EXIT!`), which is expected, @@ -202,8 +226,11 @@ flat. Before quoting, check truncation: `eval_factory_metrics.json` → at `--max-model-len 1100000`). Reference shape (reviewed golden, BF16 Nano 3.5, 1M): `pass@1 = 26.91` (2/4/8 -needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, parallelism 256, 4 nodes / -4 instances. Use it to sanity-check shape, not as a bar for another model — a +needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, 0 failures, 4 nodes / 4 +instances, Gym `a431501a`, toolchain `0.2.7+20260722.eb3ddf2a` — the sign-off run +recorded in the upstream `manifest.yaml`. That run used `parallelism: 256`; the +canonical `bench.yaml` has since moved to 512, which is a throughput change, not a +scoring one. Use the shape to sanity-check, not as a bar for another model — a rollout count well below 2363 (full runs only — a `++limit` canary is expected to be short) means tasks were lost (e.g. a walltime resume) and the score covers fewer tasks than the reference. diff --git a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md index f935e50ef57..b8ba53f2eba 100644 --- a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md +++ b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md @@ -3,14 +3,23 @@ GDPVal is currently validated with the **0.2.6 `nel` launcher** as a `nemo_gym` task, but it is mechanically unlike the `aa/` nemo-skills tasks: the Stirrup agent produces office/PDF **deliverables** in a per-task **Apptainer** code-exec sandbox, -a pairwise/rubric **judge** (Gemini 3.1 Pro) scores them, and NeMo Gym is pulled -and run **inline in the eval container** (`install_on_the_fly`) via +a rubric judge or a **3-member pairwise judge panel** scores them, and NeMo Gym is +pulled and run **inline in the eval container** (`install_on_the_fly`) via `ng_prepare_benchmark` + `ng_e2e_collect_rollouts`. This file is the shared machinery; the config template is `recipes/examples/gym/example_gdpval.yaml` and the per-task pointer is `recipes/tasks/gym/gdpval.md`. The gym bootstrap machinery described here is shared with MRCR (`recipes/tasks/gym/mrcr.md`) — fixes here apply to both examples. +**Upstream this benchmark is `GDPVal-AA-V2`** (`configs/benchmarks/gdpval-aa-v2/` → +`configs/benchmarks/gym/gdpval/{bench,refs}.yaml` in nvidia-eval-factory-benchmarking, +`dl/JoC/competitive_evaluation/…`). v2 is a different measurement from the v1 configs +this skill previously tracked: a rebuilt sandbox aligned to the **published AA GDPval-AA +v2 environment**, a refreshed 12-model reference ladder, an anchored multi-stage ELO fit, +and stage-scoped metric keys. Do not compare a v2 number to a v1 one. Re-read the +upstream `bench.yaml` + `refs.yaml` before any scored run — the reference ELO anchors +move with the AA board. + Always invoke GDPVal through the pinned wrapper, even if `nel` is already on PATH: ```bash @@ -59,15 +68,50 @@ versions within a baseline-versus-candidate comparison. | Policy model (under test) | your self-deployed vLLM endpoint (SLURM GPU node) — or an external endpoint | | NeMo Gym + Stirrup agent orchestration | inside the **eval** container (`nemo_gym` task), pulled via `install_on_the_fly` | | Per-task code-exec | **Apptainer SIF** launched by the agent inside the eval container | -| Judge (pairwise/rubric) | external OpenAI-compatible endpoint (`gdpval_judge`, e.g. Gemini 3.1 Pro) | +| Judge (pairwise/rubric) | external OpenAI-compatible endpoint (`gdpval_judge`); one model in rubric mode, a sampled 3-member panel in comparison mode | | Agent web search | Tavily (`TAVILY_API_KEY`) | +The canonical comparison-mode config also reaches two **auxiliary** endpoints through the +same proxy pattern — an `allenai/wildguard` safety server and a `Qwen3-235B` helper — each +with its own key (`COMP_EVAL_NVCF_SERVICE_KEY`, `INTEGRATE_NVAPI_KEY`). They are part of the +eval-config Step 3c conversion, not of the rubric template. + +## Three things that move together + +The Gym pin, the eval image's Python, and the SIF are **one unit**. The canonical pin +`df3e201d942f6397def1ec8c10037d29115fcbbd` declares `requires-python >= 3.13.14`, so it +must run on a py3.13 gym runtime image with +`extra.nemo_gym.runtime_python: /usr/local/bin/python3.13`, and it must mount the SIF +built from *its own* `gdpval.def` (`python-3.13.gdpval.gym-80e4fc.sif`). Move one and you +must move all three: + +| | Canonical (GDPval-AA v2) | +|---|---| +| Gym `install_on_the_fly.commit` | `df3e201d942f6397def1ec8c10037d29115fcbbd` | +| eval container | a py3.13 gym runtime image (internal `gym_runtime_py313`, CPython 3.13.14) | +| `runtime_python` | `/usr/local/bin/python3.13` — overrides the model common's py3.12 default | +| SIF | `python-3.13.gdpval.gym-80e4fc.sif` | + +A py3.12 image with this pin fails at sub-venv creation (loud). An old SIF with this pin +degrades deliverables inside the sandbox (silent). Those are the two failure shapes. + ## Apptainer SIF sandbox The Stirrup agent runs each task's generated code in an Apptainer SIF, bind-mounted at **exactly** the path `GDPVAL_CONTAINER_PATH` names (template: -`/gdpval/sif/python-3.13.gdpval.sif`). Missing or misnamed → the agent **silently** -runs code-exec unsandboxed; the run "succeeds" but the numbers aren't comparable. +`/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif`). Missing or misnamed → the agent +**silently** runs code-exec unsandboxed; the run "succeeds" but the numbers aren't +comparable. + +**What gym-80e4fc is.** The first sandbox aligned to the environment AA publishes for +GDPval-AA v2: Debian trixie (not the `python:3.13` docker image, whose 3.13.15 differs +from the closure's pinned 3.13.5 — the def *asserts* 3.13.5 at build time and fails +otherwise), the 419 published Python pins installed verbatim from the vendored freeze, +the 762-package apt closure, and the Calibri/Cambria **metric-substitute fonts** earlier +images lacked — whose absence had been reflowing every deliverable written in those +fonts. It deliberately ships **no** torch/keras/jax; the agent prompt used to advertise +them and no longer does. The name carries no architecture: build natively per cluster and +read the `BUILD_INFO` file beside a site-provided image before copying one anywhere. **If your site provides a prebuilt SIF, use it** — a self-built one resolves its pip stack at *your* build time and can drift from the sandbox a published reference set @@ -81,17 +125,22 @@ srun -p cpu -t 01:00:00 --pty "$SKILL_DIR/scripts/gdpval-sif.sh" # uses $GDPVA ``` `gdpval-sif.sh` is idempotent (flock-guarded, atomic): it builds from `gdpval.def` at -the pinned commit if absent and is a no-op once present. It needs +the pinned commit if absent and is a no-op once present. The v2 def declares a +**`%files` section** (the pin list, the apt closure, the arm64 exclusions, the sandbox +verifier), and apptainer resolves those sources against the *build's CWD* — so the def +can no longer be built on its own. The script stages the def plus every source named in +its `%files` block into a temp dir and builds from inside it; a def that adds a file +keeps working because the list is parsed, not hardcoded. It needs apptainer/singularity with unprivileged-build support and network egress — run it on a login or CPU node, **not** inside the eval job. The eval image doesn't ship apptainer, so the config's `pre_cmd` installs the **runtime** (arch-aware: use the Ubuntu PPA, not an amd64 `.deb` — most Blackwell/Grace clusters are aarch64), which needs `NEMO_EVALUATOR_TRUST_PRE_CMD=1`. -**The SIF is versioned with the Gym repo.** `gdpval.def` changes across commits (e.g. -`2502893977` → `049b1fd0` moved python 3.12 → 3.13 and added TeX Live, playwright, -polars, geospatial), and the newer agent's prompt advertises that richer runtime. So -when you bump `install_on_the_fly.commit`, **diff the def at the two commits** +**The SIF is versioned with the Gym repo.** `gdpval.def` changes across commits (the +v1→v2 move rebuilt it wholesale onto the published AA closure; an earlier one moved +python 3.12 → 3.13), and the agent's prompt describes whatever runtime the def +installs. So when you bump `install_on_the_fly.commit`, **diff the def at the two commits** (`raw.githubusercontent.com/NVIDIA-NeMo/Gym//responses_api_agents/stirrup_agent/containers/gdpval.def`); if it changed, rebuild to a **new version-tagged filename** (`GDPVAL_SIF_NAME=… gdpval-sif.sh --commit `) and repoint `GDPVAL_CONTAINER_PATH`. Running a new gym on an old SIF makes the @@ -158,21 +207,85 @@ self-deploys single-node vLLM, which is fine for a canary or a small policy. For + **`rubric`** (template default) — judge scores each deliverable against its rubric. 0–1 reward, **no ELO** (undefined without an opponent). Runs on the public gym image. + **`comparison`** — pairwise vs anchored reference deliverables; the **only** mode - yielding the AA-comparable `normalized_elo`. It is a conversion, not a flag flip: - it needs a reference set, a gym image whose Gym has the `reference_models` map, ref - mounts on **both** deployment and evaluation, and multistage overrides. Setting + yielding the AA-comparable `normalized_elo`, and the mode GDPVal-AA-V2 is defined in + upstream. It is a conversion, not a flag flip: it needs the reference set, a py3.13 + gym image whose Gym has the `reference_models` map, ref mounts on **both** deployment + and evaluation, the judge panel, and the multistage block. Setting `reward_mode=comparison` alone exits at startup with `reward_mode=comparison requires reference_deliverables_dir to be set`, surfaced only as `Process gdpval_resources_server finished unexpectedly!`. NVIDIA-internal: `modelopttools:eval-config` Step 3c is the conversion checklist. +### Comparison mode: the reference ladder + +`refs.yaml` is the single source of truth — each reference is one stanza (ELO anchor + +deliverables dir), and both the mounts and the `++…reference_models.*` overrides are +derived from it, so a path or anchor is edited in exactly one place. + +**12 active references**, spanning ELO 274 → 1468 (deepseek_v4_flash_0731 1468, +deepseek_v4_pro 1223, inkling_small 1191, glm51_fp8 1181, kimi_k26 1115, +nemotron3_ultra 1091, qwen36_35b 992, qwen35_122b 925, qwen35_397b 905, gptoss_120b 745, +gemma4_26b 713, qwen3_30b_thinking 274). Two are **deliberately disabled** and must not be +re-enabled casually: + ++ `human_gold` — 35/220 tasks ship no gold deliverable upstream, which the judge scores + as free candidate wins, and the panel rates gold ~782 against a 1000 anchor. ++ `gptoss_20b` — 23% empty deliverables, same free-win problem. + +**Anchors are held FIXED by the Bradley-Terry fit, so a stale anchor shifts every +candidate score by roughly the anchor error.** The current set was read off the AA live +board on 2026-09-09 (2026-09-11 for the three additions); the previous set sat 30–76 +points high. Match the model **variant and collection date**, not just the family — AA +scores e.g. "DeepSeek V4 Pro (Reasoning, Max Effort)" separately from "DeepSeek V4 Pro +0813". Adding a reference means staging its deliverables under the refs root *first*: +the mount fails the run if the source path is missing. + +### Comparison mode: multistage + +`_gdpval_stages.cli_params` is the one place the stage shape is defined: + +```text +++multistage.enabled=true +++multistage.retry_inprocess=true +++multistage.stages='[{num_tasks: 45, partial_completion: {min_success_fraction: 0.9, + min_per_reference_success_fraction: 0.5, min_successful_rows_per_reference: 1, + waivable_failure_classes: [timeout_exceeded, transient], tolerate_unresolved: true}}, + {num_tasks: 220, num_models: 4}]' +++num_repeats=1 +``` + +Stage 1 samples **45 tasks against all references**; stage 2 judges the full **220** against +the **4 references nearest the stage-1 estimate**. Things to know before touching it: + ++ **`partial_completion` is not optional hardening.** Without it the orchestrator requires + *every* planned non-final rollout to carry usable battle evidence, so a single unresolved + rollout — one reaped job, one judge 400 — kills stage 2 after stage 1 has already been + paid for. That happened to real runs. `legitimate` failures stay non-waivable. ++ **The resume journal is fingerprinted on the stage shape**, so changing stages invalidates + in-flight runs. `num_repeats` is *not* fingerprinted — never flip it mid-campaign. ++ **There is no per-stage repeats key.** One inside a stage dict is silently ignored; + `++num_repeats` is rollout-collection level and applies to every stage. ++ **Never combine `++limit` with multistage.** It truncates the loaded dataset while stages + sample from the full distribution → 0 rollouts. The canonical config sets + `allow_limit_samples: false` so the launcher-level limiter cannot be believed either. ++ `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=3` bounds the in-process retry (`retry_inprocess`) at two + passes. The cap alone resolves nothing — `retry_inprocess` is what re-dispatches. + ## Judge -Rubric mode uses a single judge. **Comparison mode uses a 3-member panel** — -`openai/gpt-5.5`, `gcp/google/gemini-3.1-pro-preview`, -`aws/anthropic/bedrock-claude-opus-4-8` — one **sampled per trial**, all routed -through the single `gdpval_judge_model` proxy (`` from `.env`). -`++...judge_sampling_seed=42` makes that sampling reproducible. +Rubric mode uses a single judge. **Comparison mode uses a 3-member panel**, one +**sampled per trial**, all routed through the single `gdpval_judge_model` proxy +(`` from `.env`). `++...judge_sampling_seed=42` makes that sampling +reproducible, and `++...num_comparison_trials=4` sets the trials per rollout. + +**Each member carries its own generation overrides — copy them verbatim, they are part +of the methodology:** + +| Member | `model` | `create_params_overrides` | +|---|---|---| +| gpt-5.5 | `openai/openai/gpt-5.5` | `{reasoning_effort: medium}` | +| gemini-3.1-pro | `gcp/google/gemini-3.1-pro-preview` | `{extra_body: {reasoning_effort: high}}`, `handles_audio_video: true` | +| claude-opus-4.8 | `aws/anthropic/bedrock-claude-opus-4-8` | `{extra_body: {thinking: {type: enabled}}}` | + **Inject the key's VALUE, not its name:** `openai_api_key=$INFERENCE_API_KEY`. Passing an env-var *name* (e.g. via a `${...api_key}` interpolation that resolves to @@ -181,9 +294,10 @@ through the single `gdpval_judge_model` proxy (`` from `.en not a config error. One key covers all three panel members. + **Do not set `judge_responses_create_params_overrides.model`.** Pinning a model collapses the panel to a single judge, silently changing the scoring methodology. -+ **Throttles:** judge `max_concurrent_requests=10` and Stirrup `concurrency=220` are - the golden values — the judge rate-limits long before the served model does, so raise - these only after the judge logs are clean of 429s. ++ **Throttles:** judge `max_concurrent_requests=10`, Stirrup `concurrency=220` and + `preconvert_max_concurrent=30` are the canonical values — the judge rate-limits long + before the served model does, so raise these only after the judge logs are clean of + 429s. ## Preflight — what NEL validates, and what it does NOT @@ -226,7 +340,10 @@ mount source, and `raise ValueError` listing the missing ones **before** any | `GDPVAL_CONTAINER_PATH` | lit | SIF path — must equal the SIF bind-mount target | | `GDPVAL_REF_FILES_DIR` | lit:/gdpval_ref_files | shared-FS ref-file staging (node-local /tmp breaks multi-node Ray) | | `PERSIST_DELIVERABLES_DIR` | lit | where deliverables persist (see MLflow note) | -| `GDPVAL_MAX_TURNS` | lit (optional) | Stirrup turn cap (default 100; golden uses 250) | +| `GDPVAL_MAX_TURNS` | lit (optional) | Stirrup turn cap (default 100; canonical uses 250) | +| `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` | lit:3 | bounds Gym's in-process rollout retry at two passes | +| `JUDGE_API_KEY` | host (comparison) | upstream splits judge auth from `INFERENCE_API_KEY`; this template reuses the latter for both | +| `INTEGRATE_NVAPI_KEY` / `COMP_EVAL_NVCF_SERVICE_KEY` | host (comparison) | auxiliary Qwen3-235B / wildguard endpoints in the canonical config | | `NEL_INVOCATION_ID` | runtime | stable run id assigned by the validated launcher; do not use `SLURM_JOB_ID` | `INFERENCE_JUDGE_URL` is the judge host — config (from `.env`), substituted as the @@ -256,9 +373,10 @@ nemo_gym.gdpval`). ## num_repeats -**Use 1.** Both current goldens do, set with a top-level `++num_repeats=1` — it -works, and recent Gym pins already ship `num_repeats: 1` in -`benchmarks/gdpval/config.yaml`, so no `sed` patching is needed. +**Use 1.** The canonical config sets it with a top-level `++num_repeats=1` inside the +multistage block, and recent Gym pins already ship `num_repeats: 1` in +`benchmarks/gdpval/config.yaml`, so no `sed` patching is needed. It applies to every +stage — there is no per-stage repeats key (see "Comparison mode: multistage"). Historical only: pre-multistage single-reference configs used 2 (220 × 2 = 440 rollouts) and patched it with `sed` because the per-dataset key could not be set diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 8e009098d9f..17a0c7fcf18 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -52,6 +52,14 @@ prints the resolved spec, so check there if a run behaves unexpectedly. (SWE-bench), and `${MLFLOW_TRACKING_URI}` from `.env`. Run **`modelopttools:eval-config`** (Step 3b) to write them — it holds the canonical values, arch/region rules, and points to the per-benchmark `bench.yaml` source of truth. +- **`HARBOR_ECS_REGION` selects the sandbox region** (`sandbox.region` + + `cluster.container_env.AWS_DEFAULT_REGION`); upstream defaults it to `us-east-1` for + TB2.1 and `us-east-2` for SWE-bench. The two then treat the ECR differently: TB2.1's + repo **name tracks the region** + (`463701203462.dkr.ecr..amazonaws.com/harbor-`), while SWE-bench's is + pinned to `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench` whatever the + sandbox region is. So changing the region changes the TB2.1 ECR and not the SWE-bench + one — keep `${HARBOR_*_ECR_REPOSITORY}` in `.env` consistent with it. - `set -a && source .env && set +a` before running so `${VAR}` resolves. ## Architecture — where each piece runs @@ -154,9 +162,13 @@ with its own `run_id`, copying the shared `services:` block. catch. This pair is safe because it is what the TB2.1 + SWE-bench parity runs executed, not by construction. **Re-canary whenever either side moves**, and treat a post-allocation `ValidationError` mentioning an unknown/missing config field as skew, not a config bug. -- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch). Re-check - against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which is the - pin and does move. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. +- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch; + `gitlab-master.nvidia.com/dl/joc/competitive_evaluation/nemo-evaluator-next:0.5.0.1-harbor`). + Re-check against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which + is the single pin for TB2.1 + SWE-bench Verified + SWE-bench Multilingual and does move. + It also carries `nemo_evaluator_next_version`, which the canonical configs echo into the + MLflow tag of the same name — tag your runs with it too, so a score traces back to the + image that produced it. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). - **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600). A smaller proxy timeout silently truncates long agent turns. @@ -178,7 +190,16 @@ with its own `run_id`, copying the shared `services:` block. for the life of the run, full request + response bodies, and the whole list is re-serialized on each write. A long agentic run that is 400ing or rate-limiting (the failure this dumps diagnose) grows the proxy without bound — exactly the run you can least afford to lose. - Chain position is per benchmark (last for TB2.1, first for SWE-bench). + It is **first** in the chain for both TB2.1 and SWE-bench (it used to be last for TB2.1). +- **Interceptor lists replace wholesale on merge, they do not append.** A leaf that adds + one interceptor must restate the whole chain from the model fragment + (`http_pairs_dump` → `drop_params` → `consolidate_system` → `reasoning` → + `reasoning_replay`, plus SWE-bench's `system_message` + `turn_counter`). Half a chain is + a silent scoring change, not an error. +- **`proxy.model_traffic.capture_request_body: true` belongs on the service leaf** + (`services..proxy.model_traffic`), never in a shared/benchmark-level block: a + shared `services:` entry is alias-only with no `type`, which fails the service + discriminator in every other model's composed config. - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`. @@ -188,7 +209,9 @@ with its own `run_id`, copying the shared `services:` block. - **MLflow export — config + a post-run push.** Add `output.export: [mlflow]` + `export_config.mlflow` (hardcode `experiment_name: /` — `${USER}=root` in-container; tags `framework`/`model`/`temperature`/`top_p` + `checkpoint_path`/`benchmark` - for dashboard attribution). `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` + for dashboard attribution; canonical also sets `task_name: `, + `nemo-evaluator-next-version`, and `log_config_params: true`). + `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` Step 3b — canonical `mlflow.frontier-evals.nvidia.com`; **not** the `mlflow-nemo-evaluator` alias, whose 308 strips `/api/...` → 405). **SLURM does NOT auto-export** — push after the run with `nel-next.sh mlflow-push` (Run flow), which resolves the var and falls back diff --git a/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh b/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh index 28a19e534f6..f712127d9d9 100755 --- a/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh +++ b/plugins/modelopt/skills/evaluation/scripts/gdpval-sif.sh @@ -25,8 +25,9 @@ # "$SKILL_DIR/scripts/gdpval-sif.sh" [] [--commit ] [--force|--check] # Persistent path on the target cluster's shared FS. # DEFAULTS to $GDPVAL_SIF_DIR (from .env) when omitted. A -# directory -> /$GDPVAL_SIF_NAME (default python-3.13.gdpval.sif, -# matching the example config); a *.sif path +# directory -> /$GDPVAL_SIF_NAME (default +# python-3.13.gdpval.gym-80e4fc.sif, matching the example config +# and the canonical GDPVAL_CONTAINER_PATH); a *.sif path # is used verbatim. Bind-mount this SAME dir into the eval # container at /gdpval/sif (see recipes/examples/gym/). # --commit NeMo Gym commit whose gdpval.def to build. Keep in sync @@ -51,8 +52,12 @@ set -euo pipefail # Keep GDPVAL_GYM_COMMIT in sync with install_on_the_fly.commit in the config. -GDPVAL_GYM_COMMIT="${GDPVAL_GYM_COMMIT:-dd41196f620f2af99947d776cbe5da9439d2a08d}" # pragma: allowlist secret -GDPVAL_SIF_NAME="${GDPVAL_SIF_NAME:-python-3.13.gdpval.sif}" +# df3e201d is the GDPval-AA v2 pin: its gdpval.def is the one the canonical +# gym-80e4fc image was built from (Debian trixie, python3.13=3.13.5 asserted at build +# time, the 419 published AA package pins, Office metric-substitute fonts, and no +# torch/keras/jax). Bump this and GDPVAL_SIF_NAME together. +GDPVAL_GYM_COMMIT="${GDPVAL_GYM_COMMIT:-df3e201d942f6397def1ec8c10037d29115fcbbd}" # pragma: allowlist secret +GDPVAL_SIF_NAME="${GDPVAL_SIF_NAME:-python-3.13.gdpval.gym-80e4fc.sif}" APPTAINER_BIN="${APPTAINER_BIN:-}" _log() { printf '\033[2m %s\033[0m\n' "$*" >&2; } @@ -128,11 +133,19 @@ fi [[ -n "$APPTAINER_BIN" ]] || _die "apptainer/singularity not found on PATH. Run on a node that has it \ (e.g. 'module load apptainer', or inside the eval image). This script does NOT copy a SIF from another cluster." -def_url="https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/${GDPVAL_GYM_COMMIT}/responses_api_agents/stirrup_agent/containers/gdpval.def" +def_base="https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/${GDPVAL_GYM_COMMIT}/responses_api_agents/stirrup_agent/containers" +# apptainer resolves %files SOURCES relative to the build's CWD, so the def cannot be +# built alone: stage it with its siblings and build from inside that dir. +build_dir="${sif_dir}/.gdpval-build.$$" tmp="${sif_dir}/.build.$$.${GDPVAL_SIF_NAME}" -def_local="${sif_dir}/.gdpval.$$.def" lock="${sif_dir}/.gdpval-sif.lock" +# _fetch — curl if available, else wget. +_fetch() { + if command -v curl >/dev/null 2>&1; then curl -fsSL "$1" -o "$2" + else wget -qO "$2" "$1"; fi +} + # --- build under a flock (double-checked) so concurrent runs don't double-build --- exec 9>"$lock" || _die "cannot open lock file: $lock" _log "acquiring build lock ($lock) ..." @@ -145,32 +158,53 @@ fi _log "building GDPVal SIF (this can take ~20-40 min)" _log " gym commit: ${GDPVAL_GYM_COMMIT}" -_log " def: ${def_url}" +_log " def: ${def_base}/gdpval.def" _log " dest: ${sif}" # Leave no temp artefacts if we are killed or exit early. $tmp is renamed on success, # so this only ever removes leftovers. -trap 'rm -f "$tmp" "$def_local"' EXIT -rm -f "$tmp" "$def_local" -# apptainer build cannot take a remote def URL as its source — fetch the def to a -# local file first, then build from it. -if command -v curl >/dev/null 2>&1; then curl -fsSL "$def_url" -o "$def_local" -else wget -qO "$def_local" "$def_url"; fi -[ -s "$def_local" ] || { rm -f "$def_local"; _die "failed to download def from $def_url"; } +trap 'rm -rf "$tmp" "$build_dir"' EXIT +rm -rf "$tmp" "$build_dir" +mkdir -p "$build_dir" || _die "cannot create build staging dir: $build_dir" + +# apptainer build cannot take a remote def URL as its source — fetch the def first. +_fetch "${def_base}/gdpval.def" "${build_dir}/gdpval.def" || true +[ -s "${build_dir}/gdpval.def" ] || _die "failed to download def from ${def_base}/gdpval.def" + +# The GDPval-AA v2 def declares %files (the published 419-package pin list, the apt +# closure, the arm64 exclusions, the sandbox verifier). Those sources resolve against +# the build CWD, so fetch every one named in the def — parsed rather than hardcoded, so +# a def that adds a file keeps working. +_files=$(awk ' + /^[[:space:]]*%files([[:space:]]|$)/ { inblk=1; next } + /^[[:space:]]*%/ { inblk=0 } + inblk && $1 !~ /^#/ && NF { print $1 } +' "${build_dir}/gdpval.def") +for _f in $_files; do + case "$_f" in + /*|*..*) _die "refusing to stage %files source outside the containers dir: $_f" ;; + esac + _log " staging %files source: $_f" + _fetch "${def_base}/${_f}" "${build_dir}/${_f}" || true + [ -s "${build_dir}/${_f}" ] || _die "def declares %files source '$_f' but it could not be fetched from ${def_base}/${_f}" +done + +# Absolute, because the build runs with CWD inside $build_dir. +_abs_tmp="$(cd "$(dirname "$tmp")" && pwd)/$(basename "$tmp")" # Prefer --fakeroot (needs an /etc/subuid entry for the build user); fall back to an # unprivileged build where fakeroot is unavailable. -if "$APPTAINER_BIN" build --fakeroot "$tmp" "$def_local"; then +if (cd "$build_dir" && "$APPTAINER_BIN" build --fakeroot "$_abs_tmp" gdpval.def); then : # A failed --fakeroot attempt can leave a partial $tmp behind, and apptainer refuses an # existing destination — clear it or the unprivileged fallback can never succeed. -elif rm -f "$tmp" && "$APPTAINER_BIN" build "$tmp" "$def_local"; then +elif rm -f "$_abs_tmp" && (cd "$build_dir" && "$APPTAINER_BIN" build "$_abs_tmp" gdpval.def); then _log "built without --fakeroot (unprivileged mode)" else - rm -f "$tmp" "$def_local" + rm -rf "$_abs_tmp" "$build_dir" _die "apptainer build failed (see output above)." fi -rm -f "$def_local" +rm -rf "$build_dir" # Atomic publish: a partial build never looks complete. -mv -f "$tmp" "$sif" || { rm -f "$tmp"; _die "failed to move built SIF into place: $sif"; } +mv -f "$_abs_tmp" "$sif" || { rm -f "$_abs_tmp"; _die "failed to move built SIF into place: $sif"; } _log "done: $sif" echo "$sif"