docs(eval): align GDPVal/TB2.1/SWE-bench/MRCR skills with upstream configs

plugins/modelopt/skills/evaluation/SKILL.md

@@ -42,7 +42,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead:
 
 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`.
 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched).
-3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block.
+3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block.
 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r <run_id> -c <cfg>` after (config-driven; see `references/nel-next.md`).
 
 Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.
@@ -65,18 +65,22 @@ tasks). If the user asks for GDPVal:
    (NVIDIA-internal: `modelopttools:eval-config` Step 3c); otherwise set
    `GDPVAL_SIF_DIR` in `.env` and build with `"$SKILL_DIR/scripts/gdpval-sif.sh"`
    (build-if-absent, no cross-cluster copy). Either way the mounted dir must contain
-   the file `GDPVAL_CONTAINER_PATH` names (template: `python-3.13.gdpval.sif`) — a
+   the file `GDPVAL_CONTAINER_PATH` names (canonical:
+   `python-3.13.gdpval.gym-80e4fc.sif`, the GDPval-AA v2 sandbox) — a
    name mismatch passes NEL's `test -d` check and the agent then silently runs
-   unsandboxed. Verify with `gdpval-sif.sh --check`. `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`,
+   unsandboxed. Verify with `gdpval-sif.sh --check`. The SIF, the Gym pin and the eval
+   image's Python move as **one unit** (reference: "Three things that move together").
+   `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`,
    `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR`, and `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the
    config has a `pre_cmd`). Thinking mode is mandatory (non-thinking loses ~86%).
 4. Run both dry-run and launch through `"$SKILL_DIR/scripts/nel-gdpval.sh"`; it
    enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale
    and avoids an unset `NEL_INVOCATION_ID` failure before client startup.
-   **`limit_samples` is inert on the gym path** (the gym runs all 220 tasks
-   regardless), so there is no cheap canary: watch the real run's first
-   ~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the
-   recipe's Canary section.
+   **`limit_samples` is inert on the GDPVal gym path** (the gym runs all 220 tasks
+   regardless; the config says so via `allow_limit_samples: false`), so there is no
+   cheap canary: watch the real run's first ~20–30 min for the SIF-sandbox line and
+   judge auth, and cancel if wrong. See the recipe's Canary section. (MRCR is the
+   exception — there `limit_samples` *does* reach the gym.)
 
 ---
 
@@ -92,6 +96,8 @@ for an "AA" request. If the user asks for MRCR:
 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it
    sets the context cap, dataset *and* metric prefix; the three are not
    comparable; set it in **both** `data_prep_params` and `collect_rollout_params`.
+   Upstream now carries 1M and 128K as two separate benchmarks with their own
+   manifests and metric keys — see the recipe.
 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus
    `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` +
    `transformers`; prepare fails without it) and

plugins/modelopt/skills/evaluation/recipes/env.example

@@ -40,6 +40,14 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 # GDPVal (nemo_gym) also reuses INFERENCE_JUDGE_URL for its pairwise judge.
 # INFERENCE_JUDGE_URL=https://<your-inference-host>/v1
 
+# GDPVal COMPARISON mode only — upstream splits judge auth from INFERENCE_API_KEY and
+# reaches two auxiliary endpoints. The rubric template in this skill reuses
+# INFERENCE_API_KEY for the judge and needs none of these; the
+# `modelopttools:eval-config` Step 3c conversion adds them.
+# JUDGE_API_KEY=                   # 3-member judge panel auth
+# INTEGRATE_NVAPI_KEY=             # Qwen3-235B helper endpoint
+# COMP_EVAL_NVCF_SERVICE_KEY=      # allenai/wildguard safety server
+
 # GDPVal (nemo_gym Stirrup agent) — agent web search. Secret; exported and read
 # by the harness. See recipes/tasks/gym/gdpval.md + references/gym-gdpval.md.
 # TAVILY_API_KEY=
@@ -50,6 +58,13 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 # /gdpval/sif. Convention: a per-user .cache dir.
 # GDPVAL_SIF_DIR=<shared-fs>/<user>/.cache/gdpval/sif
 
+# GDPVal (nemo_gym) — SIF filename inside GDPVAL_SIF_DIR. Only set this to point at a
+# site-provided image whose name differs from the canonical one; it must equal the
+# basename in the config's GDPVAL_CONTAINER_PATH, or NEL's `test -d` mount check passes
+# and the agent silently runs code-exec unsandboxed.
+# Default: python-3.13.gdpval.gym-80e4fc.sif (the GDPval-AA v2 sandbox).
+# GDPVAL_SIF_NAME=python-3.13.gdpval.gym-80e4fc.sif
+
 # GDPVal (nemo_gym) — Stirrup agent turn cap. Read at SUBMIT time from the
 # launching shell (the config uses ${oc.env:GDPVAL_MAX_TURNS,250}), so it must be
 # exported before `nel run`; setting it as a container env var has no effect.
@@ -67,6 +82,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 # AWS_ACCESS_KEY_ID=
 # AWS_SECRET_ACCESS_KEY=
 # NEL_NEXT_EVAL_IMAGE=             # harbor eval image (arch-matched) — set by eval-config
+# HARBOR_ECS_REGION=               # Fargate sandbox region. Upstream defaults: us-east-1
+#                                  # (Terminal-Bench), us-east-2 (SWE-bench). TB's ECR repo
+#                                  # NAME tracks this (harbor-<region>); SWE-bench's does not.
 # HARBOR_ECR_REPOSITORY=           # Terminal-Bench harbor ECR — set by eval-config
-# HARBOR_SWEBENCH_ECR_REPOSITORY=  # SWE-bench harbor ECR (us-west-2) — set by eval-config
+# HARBOR_SWEBENCH_ECR_REPOSITORY=  # SWE-bench harbor ECR (us-west-2, region-independent) — set by eval-config
 # MLFLOW_TRACKING_URI=             # MLflow tracking URI (canonical frontier-evals host) — set by eval-config

plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml

@@ -45,17 +45,27 @@ services:
     proxy:
       request_timeout: 3600                      # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout
       extra_body: {skip_special_tokens: false}   # add model-card sampling extras here if the card specifies them; mirror them in the export tags below
+      model_traffic: {capture_request_body: true}  # FEA-224 — set PER SERVICE, never in a shared block (an alias-only entry has no `type` and fails the service discriminator)
+      # Interceptor lists REPLACE wholesale on merge — if you add one, restate them all.
       interceptors:
+        # FEP-1104/1120 diagnostics — canonical position is FIRST (both TB2.1 and SWE-bench).
+        # Uncomment for a CANARY/debug run, drop it for the scored run: first_n caps only 200s,
+        # so every error pair (full req+res bodies) is retained in memory for the whole run and
+        # re-serialized on each write — unbounded growth exactly when the server errors.
+        # - name: http_pairs_dump
+        #   config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}   # $$ defers expansion to run time
+        # SWE-bench (OpenHands, multi-turn) inserts system_message + turn_counter HERE, before
+        # drop_params — don't lift that chain blind, see swebench_verified.md
         - name: drop_params                        # agents send max_tokens; many servers reject it
           # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped
           config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]}
-        # SWE-bench (OpenHands, multi-turn) adds turn_counter + system_message AND USES A DIFFERENT
-        # ORDER (drop_params before consolidate_system) — don't lift this chain — see swebench_verified.md
-        # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run:
-        # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for
-        # the whole run and re-serialized on each write — unbounded growth exactly when the server errors.
-        # - name: http_pairs_dump                    # canonical LAST in the chain (SWE-bench: first)
-        #   config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}   # $$ defers expansion to run time
+        - name: consolidate_system
+        # reasoning models only — normalize the reasoning field and replay it across turns.
+        # `reasoning_replay.mode` is per MODEL (think_tags default / native / both), not per
+        # benchmark. Drop both for instruct models.
+        # - name: reasoning
+        # - name: reasoning_replay
+        #   config: {mode: think_tags}
     node_pool: gpu
 
 benchmarks:
@@ -67,10 +77,12 @@ benchmarks:
       timeout_strategy: max          # canonical TB2.1; "task" = leaderboard-comparable
       agent_kwargs: {llm_kwargs: {timeout: 3600}}
     sandbox:
-      region: us-east-1              # MUST match the region in ${HARBOR_ECR_REPOSITORY} (SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY})
+      # TB2.1's ECR repo NAME tracks the region (harbor-<region>); SWE-bench's is pinned to
+      # us-west-2/harbor-swebench whatever the sandbox region is (us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}).
+      region: ${HARBOR_ECS_REGION:-us-east-1}
       ecr_repository: ${HARBOR_ECR_REPOSITORY}      # from modelopttools:eval-config
       concurrency: 50
-      log_stream_prefix: terminalbench21-???
+      log_stream_prefix: terminalbench-21-???      # canonical stem + <model>-<framework>
 
 cluster:
   type: slurm
@@ -87,7 +99,7 @@ cluster:
     HF_HOME: /cache/huggingface
     AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
     AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
-    AWS_DEFAULT_REGION: us-east-1     # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY}
+    AWS_DEFAULT_REGION: ${HARBOR_ECS_REGION:-us-east-1}   # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY}
     LLM_API_KEY: "no-key-needed"
   mount_home: false
   auto_resume: true
@@ -110,4 +122,12 @@ output:
       exclude_patterns: ["shard*", "model_traffic.jsonl"]   # captured request bodies (FEA-224) stay in the run dir
       description: ???               # '<model> | T=1.0 top_p=0.95 | <benchmark> (timeout_strategy=…) | r8'
       # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above.
-      tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'}
+      tags:
+        framework: vllm
+        model: "???"
+        checkpoint_path: "???"
+        benchmark: "???"
+        task_name: "terminal-bench-2.1"          # upstream benchmark slug (SWE-bench: swebench-verified)
+        nemo-evaluator-next-version: "0.5.0.1"   # eval_image version, so a score traces to its image
+        temperature: '1.0'
+        top_p: '0.95'

plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml

@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 # =============================================================================
-# GDPVal (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode.
+# GDPVal-AA v2 (NeMo Gym "Stirrup" agent) — STANDALONE gym eval, RUBRIC mode.
 # Self-deploys a checkpoint via vLLM on one SLURM node. One gym eval per config.
 #
 # Read recipes/tasks/gym/gdpval.md + references/gym-gdpval.md first — they
@@ -42,20 +42,25 @@ defaults:
 # reference deliverables and runs on the public gym image, so it works standalone.
 #   rubric     — standalone LLM-judge scoring, 0-1 reward per deliverable. (this template)
 #   comparison — pairwise vs anchored reference deliverables; the only mode that yields
-#                an AA-comparable ELO / win-rate. It needs a reference set, a newer gym
-#                image, and its own overrides — do NOT just flip reward_mode here.
+#                an AA-comparable ELO, and the mode the upstream GDPVal-AA-V2 benchmark
+#                is defined in. It needs the 12-reference set, a py3.13 gym image, a
+#                3-judge panel and multistage overrides — do NOT just flip reward_mode.
 # To run comparison, NVIDIA-internal users should follow `modelopttools:eval-config`
 # Step 3c, which converts this config (container override + reference_models map +
 # mounts + multistage). See references/gym-gdpval.md "Scoring modes".
 gdpval:
   reward_mode: rubric
 
-# GDPVal pairwise judge. base_url is config (from .env), not a secret, so no
-# export needed; only api_key (INFERENCE_API_KEY) is exported and read by the
-# harness. Keep the judge fixed across comparable runs.
+# GDPVal judge. base_url is config (from .env), not a secret, so no export needed;
+# only api_key (INFERENCE_API_KEY) is exported and read by the harness. Keep the judge
+# fixed across comparable runs.
+# RUBRIC mode (this template) scores with this single judge. COMPARISON mode routes a
+# 3-member panel — gpt-5.5 / gemini-3.1-pro / claude-opus-4.8, one SAMPLED PER TRIAL at
+# judge_sampling_seed=42 — through this same proxy; the panel is added by the
+# eval-config Step 3c conversion, not here. See references/gym-gdpval.md "Judge".
 gdpval_judge:
   base_url: <INFERENCE_JUDGE_URL>              # from .env (/v1 base); shared inference host
-  model: gcp/google/gemini-3.1-pro-preview     # Gemini 3.1 Pro; use an equivalent on your endpoint if needed
+  model: gcp/google/gemini-3.1-pro-preview     # rubric judge; also the panel's default member
   api_key: INFERENCE_API_KEY   # doc only; common_params injects the VALUE ($INFERENCE_API_KEY).
                                # An env-var NAME reaches the proxy as a literal -> opaque judge 500.
 
@@ -136,7 +141,13 @@ evaluation:
     TAVILY_API_KEY: host:TAVILY_API_KEY        # Stirrup agent web search
     UV_CACHE_DIR: lit:/cache/uv
     # Apptainer SIF for the Stirrup per-task code-exec sandbox (must match the mount above).
-    GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.sif
+    # gym-80e4fc is the GDPval-AA v2 sandbox: Debian trixie, python 3.13.5 asserted at
+    # build time, the 419 published AA package pins, and the Office metric-substitute
+    # fonts earlier images lacked. It is versioned WITH the Gym pin below — move both or
+    # neither ($SKILL_DIR/scripts/gdpval-sif.sh defaults to this pair).
+    GDPVAL_CONTAINER_PATH: lit:/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif
+    # Bounds Gym's in-process rollout retry. Canonical is 3 (= two retry passes).
+    NEMO_GYM_MAX_ROLLOUT_ATTEMPTS: lit:3
     # Shared-FS staging for ref files; node-local /tmp breaks multi-node Ray.
     GDPVAL_REF_FILES_DIR: lit:/gdpval_ref_files
     # Deliverables land under /results (already mounted) so they persist. The
@@ -205,28 +216,43 @@ evaluation:
       # as a non-git directory, so `install_on_the_fly.commit` below is silently
       # ignored here (the prepare step logs "/opt/Gym is not a git repo; using baked-in
       # Gym version") — the pin only applies on images where /opt/Gym IS a git repo.
+      # ON A GIT-BACKED IMAGE THE PIN APPLIES, AND IT NEEDS PYTHON >= 3.13.14. The
+      # canonical config pairs it with a py3.13 gym runtime image plus
+      # `runtime_python: /usr/local/bin/python3.13` (uncomment below). A py3.12 image
+      # + this pin fails at venv creation, not at scoring.
       container: nvcr.io/nvidia/eval-factory/nemo-gym:26.05   # pin a verified tag
       nemo_evaluator_config:
         config:
           params:
             extra:
               nemo_gym:
+                # limit_samples is INERT on the gym path; canonical says so explicitly
+                # so a "2-sample smoke test" can't be believed. Also required by
+                # multistage (comparison mode): ++limit truncates the loaded dataset
+                # while stages sample from the full distribution -> 0 rollouts.
+                allow_limit_samples: false
+                # Uncomment WITH a py3.13 gym runtime image above — the pinned Gym
+                # declares requires-python >= 3.13.14. Move image + interpreter + pin
+                # as one unit.
+                # runtime_python: /usr/local/bin/python3.13
                 install_on_the_fly:
                   url: https://github.com/NVIDIA-NeMo/Gym
                   # Gym version. BUMPING THIS REQUIRES REBUILDING THE SIF from the matching
                   # commit — gdpval.def (the sandbox) is versioned with the gym repo, and an
                   # old SIF + new gym silently degrades deliverables. Rebuild with
                   # `gdpval-sif.sh --commit <sha>` to a new name + repoint GDPVAL_CONTAINER_PATH.
                   # See references/gym-gdpval.md "Rebuild the SIF when the Gym version changes".
-                  # Current golden pin (updated GDPVal task-sampling algo, which drives
-                  # multistage stage-1 selection). Its gdpval.def is byte-identical to
-                  # 049b1fd0…, so a python-3.13 SIF built from either commit is valid —
-                  # no rebuild when moving between them.
+                  # Current canonical pin (GDPval-AA v2). Carries the AA reference-ELO
+                  # anchor refresh, the three multistage reference anchors, long-context
+                  # budgeting (completion floor 8192), and the gdpval.def the gym-80e4fc
+                  # SIF above was built from. Runs on it are NOT strictly comparable to
+                  # runs on the older dd41196f pin — the agent's prompt and generation
+                  # behaviour both changed.
                   # NOTE: this pin is INERT on the public nemo-gym image (see the
                   # `container:` comment above) — it only takes effect on an image where
                   # /opt/Gym is a git repo. Confirm via "=== NeMo Gym commit ===" + SHA
                   # in the client log before crediting it with any behaviour change.
-                  commit: dd41196f620f2af99947d776cbe5da9439d2a08d  # pragma: allowlist secret
+                  commit: df3e201d942f6397def1ec8c10037d29115fcbbd  # pragma: allowlist secret
                 command: |
                   set -ex
                   cd /opt/Gym
@@ -327,6 +353,9 @@ evaluation:
                 # No judge_responses_create_params_overrides.model here: the judge model is
                 # already set by openai_model=${gdpval_judge.model}, and pinning it a second
                 # way silently collapses comparison mode's 3-member judge panel to one judge.
+                # Comparison mode additionally appends num_comparison_trials=4,
+                # judge_sampling_seed=42, the judge_panel list, the reference_models map and
+                # the multistage block — all from eval-config Step 3c, none of them here.
                 # num_repeats=1 — both current goldens. No reference_* overrides here:
                 # the single-reference keys conflict with comparison mode's reference_models
                 # map (added by modelopttools:eval-config Step 3c).
@@ -357,7 +386,7 @@ export:
   mlflow:
     tracking_uri: ${oc.env:MLFLOW_TRACKING_URI}   # from modelopttools:eval-config
     experiment_name: ${oc.env:USER}/CHANGEME-served-model-name
-    description: 'CHANGEME-served-model-name | GDPVal rubric | T=1.0, top_p=0.95, num_repeats=1'
+    description: 'CHANGEME-served-model-name | GDPVal-AA v2 rubric | T=1.0, top_p=0.95, num_repeats=1'
     log_logs: true
     only_required: false
     tags: