From 4fdf5db9c1866a398dd9adcae0876a50402a3fbb Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Tue, 15 Sep 2026 22:37:06 +0000 Subject: [PATCH 1/6] docs: require evaluation provenance for model-card generation overrides Signed-off-by: Chad Voegele --- plugins/modelopt/skills/day0-release/SKILL.md | 10 +- plugins/modelopt/skills/evaluation/SKILL.md | 67 ++++++++----- .../recipes/examples/example_eval.yaml | 30 +++--- .../recipes/examples/example_eval_next.yaml | 15 ++- .../recipes/examples/gym/example_gdpval.yaml | 15 ++- .../recipes/examples/gym/example_mrcr.yaml | 15 ++- .../tasks/aa_next/swebench_verified.md | 2 +- .../evaluation/references/gym-gdpval.md | 9 +- .../references/model-card-research.md | 46 ++++----- .../skills/evaluation/references/nel-next.md | 13 ++- .../references/nvfp4-modelcard-sampling.md | 93 ++++++------------- .../skills/evaluation/tests/evals.json | 9 +- .../references/analyze-results.md | 8 +- 13 files changed, 169 insertions(+), 163 deletions(-) diff --git a/plugins/modelopt/skills/day0-release/SKILL.md b/plugins/modelopt/skills/day0-release/SKILL.md index ee58d078219..3076b6e122d 100644 --- a/plugins/modelopt/skills/day0-release/SKILL.md +++ b/plugins/modelopt/skills/day0-release/SKILL.md @@ -143,8 +143,14 @@ diff <(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \ Any other difference biases the comparison and invalidates the gate: a mismatched `parallelism` between the two sides was worth ~2 pp, enough to invert the sign of -the delta. If the model card splits sampling params per scenario, apply the same -split to both sides. Never set an unbounded `request_timeout` (`1e9`) — it turns a +the delta. Follow the evaluation skill's generation-parameter policy: explicit +user/task requirements first; model-card overrides only for values explicitly +used in evaluation/benchmarking of the applicable task/mode. Otherwise preserve +`config.json`, `generation_config.json`, and vLLM defaults, without family-based +or generic fallbacks. Apply any benchmark-specific split to both sides. Verify +effective sampling parity too: identical client configs can inherit different +checkpoint or evaluator defaults; omission is not explicit `null`. +Never set an unbounded `request_timeout` (`1e9`) — it turns a transient stall into a job that holds its GPUs until the wall clock kills it. Gate: diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index da57288528f..1d7e1d3a089 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -144,7 +144,7 @@ Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. 1. Read the task reference file(s). 2. Use `recipes/examples/example_eval.yaml` as the base. 3. Copy the YAML fragment(s) into `evaluation.tasks`, applying any per-task notes. -4. **MLflow auto-export is on by default** — it needs **two** pieces, both in `example_eval.yaml`: (a) the **trigger** `execution.auto_export.destinations: [mlflow]` (without it the run is *not* uploaded), and (b) the `export.mlflow` block that configures it. In the `export.mlflow` block use **literal** values for `experiment_name` / `description` / `tags` — substitute the actual `served_model_name` and sampling params. Do **not** use `${deployment.*}` / `${evaluation.*}` cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with `Interpolation key '...' not found` (`${oc.env:USER}` and `${oc.env:MLFLOW_TRACKING_URI}` are fine — they're env vars). Because these literals can't interpolate, keep the `temperature` / `top_p` / `max_new_tokens` tags **equal to** the top-level `params` and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. `tracking_uri` = `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config` (not hand-filled), and auto-export needs `execution.cpu_partition` (e.g. gcp-nrt `cpu`) — it's a separate CPU-only sbatch that GPU-only partitions reject (`Cannot find GPU specification`), silently dropping the link. Before filling `experiment_name`/`tags`, read the checkpoint's `.experiment.json` (Step 3) and carry the PTQ run's experiment name plus its `modelopt_*` tags across (Step 4). +4. **MLflow auto-export is on by default** — it needs **two** pieces, both in `example_eval.yaml`: (a) the **trigger** `execution.auto_export.destinations: [mlflow]` (without it the run is *not* uploaded), and (b) the `export.mlflow` block that configures it. In the `export.mlflow` block use **literal** values for `experiment_name` / `description` / `tags` — substitute the actual `served_model_name` and sampling params. Do **not** use `${deployment.*}` / `${evaluation.*}` cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with `Interpolation key '...' not found` (`${oc.env:USER}` and `${oc.env:MLFLOW_TRACKING_URI}` are fine — they're env vars). Because these literals can't interpolate, keep the `temperature` / `top_p` / `max_new_tokens` tags **equal to** explicit top-level `params` (label absent fields `omitted`, not an assumed effective value) and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. `tracking_uri` = `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config` (not hand-filled), and auto-export needs `execution.cpu_partition` (e.g. gcp-nrt `cpu`) — it's a separate CPU-only sbatch that GPU-only partitions reject (`Cannot find GPU specification`), silently dropping the link. Before filling `experiment_name`/`tags`, read the checkpoint's `.experiment.json` (Step 3) and carry the PTQ run's experiment name plus its `modelopt_*` tags across (Step 4). 5. Proceed to Step 3, then Step 4, then Step 7.5/8. Skip Step 2's 5-question flow. --- @@ -266,7 +266,7 @@ deployment: <... rest of cross-checked flags ...> ``` -Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `--served-model-name ${deployment.served_model_name}` (**required**; see `example_eval.yaml` for why); always `--host 0.0.0.0 --port ${deployment.port}`; use folded scalar (`>-`) for one flag per line. Example fallback `--max-model-len 131072` covers AA-LCR (~120K + 16K gen) and SciCode (≥ 65536) — prefer `config.json` / recipe value. +Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `--served-model-name ${deployment.served_model_name}` (**required**; see `example_eval.yaml` for why); always `--host 0.0.0.0 --port ${deployment.port}`; use folded scalar (`>-`) for one flag per line. Preserve the checkpoint/vLLM context default unless an explicit task requirement or verified deployment recipe requires an override; check that the model supports the requested context. For how to choose `--tensor-parallel-size` / `--data-parallel-size` / `--pipeline-parallel-size` (and EP) from the model size and your GPU count, read `references/parallelism.md` — cross-check the layout against `recipes.vllm.ai`, then adapt to the GPUs you actually have via the fit math there. @@ -300,7 +300,10 @@ Silence is not contradiction. Drop/override only when the recipe sets a differen #### Evaluation params template (top-level params) -The top-level `nemo_evaluator_config.config.params` must contain **exactly these six fields** — no `top_k` / `presence_penalty` / `repetition_penalty` / `min_p`: +Start with these operational fields in `nemo_evaluator_config.config.params`. +Add generation overrides only under the policy below; route `top_k` / +`presence_penalty` / `repetition_penalty` / `min_p` through the harness-supported +request adapter, not unsupported top-level fields. ```yaml nemo_evaluator_config: @@ -309,30 +312,44 @@ nemo_evaluator_config: parallelism: ??? # Required — size per references/parallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear request_timeout: 3600 max_retries: 10 - max_new_tokens: 65536 # see rule below - temperature: 1.0 # from model card (reasoning); adjust - top_p: 0.95 # from model card (reasoning); adjust ``` -Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere. - -**Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` / `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's "Lookup" section. - -**`temperature` / `top_p` are different: per-task overrides ARE allowed and often required.** Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives `top_p = 0.95` for agentic scenarios and `1.0` otherwise, so a single top-level `0.95` is wrong for every non-agentic task. -Set the top-level value for the majority case, override only the tasks the card calls out, and apply -the split identically to baseline and candidate. **The `export.mlflow` tags record only the -top-level values**, so note any per-task override in the run `description` — otherwise the -overridden task is reported under sampling params it did not use. - -#### `max_new_tokens` — mandatory model-card lookup - -1. **Fetch the HF model card before writing the value.** Not optional. -2. Scan for any `max_tokens` / `max_new_tokens` / "output length" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment. - **Card figures are SINGLE-TURN.** On multi-turn / agentic benchmarks the model's own answer is fed back in, so the cap must satisfy `n_turns × max_new_tokens + prompt < max_model_len`. Taking a card's headline "384K output" literally lost SciCode samples to HTTP 400; 65536 was clean. (`references/run-validation.md` already covers checking `finish_reason: length` after a run.) -3. **Consult `references/nvfp4-modelcard-sampling.md` as a reference.** Listed and in agreement → proceed with confidence. Listed and different → **the card wins**; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its `max_num_tokens` column records the card's *headline* cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row. -4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user. -5. **Forbidden:** writing `max_new_tokens: ` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence. -6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence. +#### Generation parameters — provenance and precedence + +1. **Explicit user/task requirements take precedence.** Preserve required token + budgets and sampling settings; do not replace them with model-card values. +2. **Read the full model card before deriving overrides.** Override + `max_new_tokens` / `max_tokens`, `temperature`, `top_p`, or other generation + parameters only when the card explicitly says they were used for evaluation + or benchmarking. Cite the statement and its applicable tasks/mode. General + inference recommendations, quickstarts, supported limits, and unrelated + scenarios do not qualify. +3. **Otherwise preserve checkpoint/server defaults** from `config.json`, + `generation_config.json`, and vLLM. Do not invent reasoning/non-reasoning + fallbacks or borrow values from related models. Cross-check + `references/nvfp4-modelcard-sampling.md` against the exact card; never use it + to fill silent or ambiguous fields. +4. **Verify the effective request.** Omitting a client parameter does not bypass + evaluator/task defaults; inspect the resolved config and canary requests. + Where supported, remove unintended client overrides so server defaults apply, + or explicitly carry verified checkpoint/server values. An explicit `null` + is not omission or a universal "uncapped" setting: verify its meaning in the + selected harness/server. Report unresolved defaults rather than guessing. +5. **Keep settings scoped to their evidence.** Use shared top-level values only + where applicable; use per-task overrides (including token caps) for explicit + requirements or benchmark-specific card settings. Never take the highest + value anywhere in a card as a suite-wide cap. Apply the same policy to + baseline and candidate, and verify their effective settings match. + +Record explicit values and their sources in the config. Keep MLflow tags in +sync; label omitted client fields as `omitted`, not as assumed numeric defaults, +and record verified effective values and per-task differences in the description. + +For output length, distinguish generation limits from the server's context +window. Check prompt + requested output against `max_model_len`, including +accumulated history on multi-turn tasks. Surface conflicts instead of silently +shrinking required budgets. Higher caps do not fix runaway reasoning; inspect +`finish_reason: length` per `references/run-validation.md`. #### Quantization-aware benchmark defaults diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml index 1f8faf6def3..5e115601c3d 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml @@ -99,6 +99,8 @@ deployment: # model as `/checkpoint` and eval requests 404 ("model does not exist"). # For MoE models, add `--enable-expert-parallel` to the command. # For models with custom code, add `--trust-remote-code` to the command. + # Preserve checkpoint/vLLM context defaults unless a task or verified deployment + # recipe requires --max-model-len; confirm the model supports that context. # `--model-loader-extra-config` enables multithreaded checkpoint loading (on by # default below) — cuts large-checkpoint load from tens of minutes to a few; # tune num_threads to the checkpoint size / shared-FS read bandwidth. @@ -112,7 +114,6 @@ deployment: --port ${deployment.port} --tensor-parallel-size 1 --data-parallel-size 1 - --max-model-len 131072 --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}' --max-num-batched-tokens 8192 --enable-chunked-prefill @@ -132,9 +133,12 @@ evaluation: parallelism: ??? # Number of concurrent requests per each benchmark request_timeout: 3600 max_retries: 10 - max_new_tokens: 65536 # 64K for reasoning models; use 16384 (16K) for non-reasoning; prefer model card value - temperature: 1.0 # from model card (reasoning mode); adjust per card - top_p: 0.95 # from model card (reasoning mode); adjust per card + # Add generation overrides for explicit user/task requirements first. + # Otherwise use model-card values ONLY when explicitly used for evaluation/ + # benchmarking of the applicable task/mode (cite the statement). + # Without that evidence, preserve config.json/generation_config.json/vLLM + # defaults. Omission can still activate evaluator defaults: inspect resolved + # configs and canary requests (SKILL Step 3). null is not omission. target: api_endpoint: api_key_name: DUMMY_API_KEY @@ -162,7 +166,8 @@ evaluation: thinking: true enable_thinking: true # DeepSeek V4 only: reasoning-effort axis (high|max). "max" needs - # --max-model-len >= 393216. Uncomment for DeepSeek V4. + # --max-model-len >= 393216. Uncomment only for an explicit requirement + # or an applicable, evaluation-provenanced model-card setting. # reasoning_effort: max tasks: # Reasoning (chat endpoint, short). nemo-skills GPQA on the AA 4-choice MCQ @@ -187,9 +192,10 @@ export: # CAUTION — these literals can drift. temperature / top_p / max_new_tokens are the # ONLY queryable record of the sampling config in MLflow (NEL does not log them as # run params), so keep them — but because they can't be interpolated, they MUST be - # kept EQUAL to evaluation.nemo_evaluator_config.config.params above. When you - # change the sampling params (or served_model_name), update these literals in the - # SAME edit, or MLflow will misreport the run. + # kept EQUAL to explicit config.params values above. 'omitted' means no client + # override here, NOT a verified server default; record effective values and + # per-task differences in the description after checking resolved requests. + # Update params and tags in the SAME edit, or MLflow will misreport the run. mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config (canonical frontier-evals host; NOT the -nemo-evaluator alias) # If the checkpoint holds a .experiment.json, use ITS experiment_name verbatim here so @@ -197,7 +203,7 @@ export: # would resolve. Cross-server, that makes a same-named experiment here and only # modelopt_run_url reaches the PTQ run (Step 4). experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=65536' + description: 'CHANGEME-served-model-name | generation overrides omitted; verify effective settings' log_logs: true log_artifacts: true only_required: false @@ -205,9 +211,9 @@ export: tags: framework: vllm model: CHANGEME-served-model-name - temperature: '1.0' - top_p: '0.95' - max_new_tokens: '65536' + temperature: 'omitted' + top_p: 'omitted' + max_new_tokens: 'omitted' # From the checkpoint's .experiment.json when present; drop when absent. Keep the # quotes (a bare 20260910 becomes a date) and skip any value containing ${ -- quoting # does not stop OmegaConf resolving it. diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 9463c653062..873e06f4aeb 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -41,10 +41,14 @@ services: container_mounts: # source dirs MUST pre-exist (pyxis won't create them): ssh 'mkdir -p //.cache/{vllm,huggingface}' - ???:/cache/vllm - ???:/cache/huggingface - generation: {temperature: 1.0, top_p: 0.95} # from model card (reasoning mode); adjust per card — mandatory lookup (references/model-card-research.md), same as 0.2.6 + # Add generation fields for explicit user/task requirements first; otherwise + # only for model-card settings explicitly used for evaluation/benchmarking. + # Preserve checkpoint/vLLM defaults absent that evidence. Omission can still + # activate evaluator/agent defaults; inspect requests. null is not omission. + generation: {} # see SKILL Step 3; mirror explicit values in tags proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout - extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below + extra_body: {skip_special_tokens: false} # add only requirement- or evaluation-provenanced generation extras; mirror in tags interceptors: - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped @@ -108,6 +112,7 @@ output: log_config_params: true copy_logs: true exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir - description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' - # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. - tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} + description: ??? # ' | verified effective sampling + sources | (timeout_strategy=…) | r8' + # model/checkpoint_path/benchmark drive dashboard attribution. Sampling tags + # mirror explicit generation fields; 'omitted' does NOT assert effective defaults. + tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: 'omitted', top_p: 'omitted'} diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml index 6e5eef250fa..73b0e991cc5 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml @@ -167,8 +167,13 @@ evaluation: nemo_evaluator_config: config: params: - temperature: 1.0 - top_p: 0.95 + # Resolve per SKILL Step 3: explicit user/task requirements, then applicable + # evaluation-provenanced card values, otherwise verified checkpoint/vLLM defaults. + # These fields feed collect_rollout_params below; if omitting them, remove + # those overrides too and verify Gym/evaluator defaults in actual requests. + # Do not use null as a substitute for omission. Update export tags together. + temperature: ??? + top_p: ??? parallelism: 16384 # gym-internal concurrency, NOT a model-server cap request_timeout: 36000 # gym rollouts are long-running max_retries: 10 @@ -357,12 +362,12 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | GDPVal rubric | T=1.0, top_p=0.95, num_repeats=1' + description: 'CHANGEME-served-model-name | GDPVal rubric | T=CHANGEME, top_p=CHANGEME, num_repeats=1' log_logs: true only_required: false tags: framework: vllm model: CHANGEME-served-model-name benchmark: nemo_gym.gdpval - temperature: '1.0' - top_p: '0.95' + temperature: ??? + top_p: ??? diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 01c1c9a42c4..986cc4e4e91 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -129,8 +129,13 @@ evaluation: nemo_evaluator_config: config: params: - temperature: 1.0 - top_p: 0.95 + # Resolve per SKILL Step 3: explicit user/task requirements, then applicable + # evaluation-provenanced card values, otherwise verified checkpoint/vLLM defaults. + # These fields feed collect_rollout_params below; if omitting them, remove + # those overrides too and verify Gym/evaluator defaults in actual requests. + # Do not use null as a substitute for omission. Update export tags together. + temperature: ??? + top_p: ??? # Gym client concurrency, TOTAL across instances — not a per-server cap. parallelism: 256 request_timeout: 36000 # 1M-token prefills are slow @@ -308,12 +313,12 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | MRCR 1M | T=1.0, top_p=0.95, num_repeats=1' + description: 'CHANGEME-served-model-name | MRCR 1M | T=CHANGEME, top_p=CHANGEME, num_repeats=1' log_logs: true only_required: false tags: framework: vllm model: CHANGEME-served-model-name benchmark: nemo_gym.mrcr_n3_1m # CHANGEME with the variant (see VARIANT SELECTOR) - temperature: '1.0' - top_p: '0.95' + temperature: ??? + top_p: ??? diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index bbd70551cb9..e9095502ec2 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -96,7 +96,7 @@ from the scored config. ```yaml proxy: request_timeout: 3600 - extra_body: {skip_special_tokens: false} # add model-card sampling extras if the card sets them + extra_body: {skip_special_tokens: false} # generation extras follow SKILL Step 3's provenance/precedence policy model_traffic: {capture_request_body: true} # FEA-224; adds the upstream request body to the traffic capture that is ALREADY ON by default interceptors: # - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} # canary only diff --git a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md index f935e50ef57..fd573d2abb8 100644 --- a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md +++ b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md @@ -143,10 +143,11 @@ self-deploys single-node vLLM, which is fine for a canary or a small policy. For SKILL Step 3/4's `ceil(parallelism / DP)` rule assumes `parallelism` is the in-flight request count; on the gym path it is not, and applying it literally gives an absurd cap. Use `ceil(stirrup_agent.concurrency / DP)` — e.g. 220 / DP 4 → 55, round to 64. -+ **`max_new_tokens`:** the reviewed golden **does** set it alongside the adapter's - `params_to_remove: [max_tokens, max_completion_tokens]`, so do the Step 3 model-card - lookup as normal. The template omits it (five params) because the adapter strips the - per-request cap anyway; adding it back matches the golden and is harmless. ++ **`max_new_tokens`:** the template omits it and the adapter removes + `max_tokens` / `max_completion_tokens`. Follow Step 3's generation provenance + policy, not a golden config's incidental value. Omission can activate evaluator + defaults; inspect requests after adapters. If a user/task requires a cap, + ensure it reaches the server rather than being stripped. + **Match `temperature` / `top_p` to whatever the reference deliverables were generated with.** A pairwise ELO compares your deliverables against theirs, so a sampling difference lands in the score as if it were a quality difference. diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index 80db5cce28b..2cd13e25dd0 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -2,38 +2,32 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it carefully, the FULL text, the devil is in the details. Extract ALL relevant configurations: -- Sampling params (`temperature`, `top_p`) - - **Only trust a sentence that ties the values to the benchmarks** ("Benchmarked - with…", "…were evaluated with…", "We evaluate the model using…") or a - "Recommended Sampling" row. The `SamplingParams(temperature=0.8, top_p=0.95)` - in a card's TensorRT-LLM/vLLM quickstart snippet is boilerplate copied - verbatim across unrelated models — **never** read eval settings out of it. - - **Then cross-check `nvfp4-modelcard-sampling.md`** — published - `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints - under `huggingface.co/nvidia` that disclose them, grouped by family. - Required for any NVFP4 checkpoint or same-family sibling; it is also the - best source of a default when the card is silent. Pre-2026 releases are out - of scope there — read their cards. -- Context length (`deployment.extra_args: "--max-model-len "`) -- **Output length (`max_new_tokens`) — mandatory extraction.** Scan the - card for any `max_tokens` / `max_new_tokens` / "output length" - recommendation. Cards often list two values (e.g., Qwen3.x: `32768` - thinking-general + `81920` math/coding). **Pick the highest value** and - apply at the top level (no per-task overrides). If the card is genuinely - silent on output length, note that explicitly and fall back to the - generic default (64K reasoning / 16K non-reasoning) — never write a - config with "card not yet checked" + generic default. Check - `nvfp4-modelcard-sampling.md` for the model or its family before falling - back; a same-family published cap beats the generic default. See SKILL.md - Step 3 "`max_new_tokens` — pick a single top-level value" for the full - rule. +- **Generation params (`temperature`, `top_p`, `max_new_tokens`, etc.).** + Explicit user/task requirements take precedence. Derive model-card overrides + only from statements tying each value to evaluation/benchmarking ("Benchmarked + with…", "…were evaluated with…"). General "Recommended Sampling" rows, + quickstarts, and supported output/context limits are not evaluation evidence. + Cite the statement and scope each override to the stated benchmark/mode; + never choose the highest value across unrelated scenarios. + - Cross-check `nvfp4-modelcard-sampling.md` against the exact card, per field. + Do not fill gaps from the table or same-family models. + - If evidence is absent or ambiguous, preserve `config.json`, + `generation_config.json`, and vLLM defaults; no generic token-budget fallback. + Omitting client fields can still activate evaluator/task defaults. Inspect + resolved configs and canary requests, and remove unintended overrides or + carry verified checkpoint/server values through the supported harness path. + Do not substitute `null` for omission without verifying its semantics. + See SKILL.md Step 3 "Generation parameters — provenance and precedence". +- Context length (vLLM `--max-model-len`): preserve the checkpoint/server default + unless the task or verified deployment recipe requires an override; check + prompt + output capacity separately from generation limits. - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed) - Reasoning config (if applicable): - reasoning on/off: use either: - `adapter_config.custom_system_prompt` (like `/think`, `/no_think`) and no `adapter_config.params_to_add` (leave `params_to_add` unrelated to reasoning untouched) - `adapter_config.params_to_add` for payload modifier (like `"chat_template_kwargs": {"enable_thinking": true/false}`) and no `adapter_config.custom_system_prompt` and `adapter_config.use_system_prompt: false` (leave `custom_system_prompt` and `use_system_prompt` unrelated to reasoning untouched). - **The `chat_template_kwargs` toggle key drifts across model generations — read the card / `chat_template.jinja`, don't extrapolate, and set only the one key the model uses.** Known: `enable_thinking` (Qwen3.5/3.6, GLM 5.1 — note GLM-4.x used `thinking`+`/nothink`); `thinking` (Kimi K2.6 — renamed from K2.5's `enable_thinking`; DeepSeek V3.2/V4 — Python encoder, not Jinja, so an unused kwarg can error rather than be ignored). - - reasoning effort/budget (if configurable, e.g. DeepSeek V4 `reasoning_effort`): **default to `max`** (the highest effort the card documents), honoring any tied requirement (e.g. V4 Think Max needs `--max-model-len >= 393216`). AskUserQuestion only if the user signals a cost/latency preference. + - reasoning effort/budget: apply the generation-parameter provenance policy above, not the highest documented effort by default. Honor tied deployment requirements (e.g. DeepSeek V4 Think Max needs `--max-model-len >= 393216`). - etc. - Deployment-specific `extra_args` for vLLM/SGLang (look for the vLLM/SGLang deployment command) - Deployment-specific vLLM/SGLang versions (by default we use latest docker images, but you can control it with `deployment.image` e.g. vLLM above `vllm/vllm-openai:v0.11.0` stopped supporting `rope-scaling` arg used by Qwen models) diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 8e009098d9f..a21a5403707 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -131,10 +131,15 @@ SKILL.md Step 3 (same vLLM). The 0.2.6 `command:` maps to structured `services.< | `image:` (bump to the model's recipe min; NVFP4 on sm_103 → CUDA-13 build, see Step 3) | `image:` (serving image, ≠ `eval_image`) | Size TP/DP + backend defaults (`--max-num-seqs = ceil(max_parallelism/DP)`, MoE -`--enable-expert-parallel`, …) per `references/parallelism.md` + Step 3. **Sampling -is a mandatory model-card lookup** (Step 3 / `references/model-card-research.md`, -never generic defaults): `generation.temperature`/`top_p` (+`max_tokens` if the card -caps output) and `proxy.extra_body` for any card extras (`skip_special_tokens`, thinking toggles). +`--enable-expert-parallel`, …) per `references/parallelism.md` + Step 3. +**Generation overrides follow Step 3's provenance/precedence policy:** explicit +user/task requirements first, then only card values explicitly used for the +applicable evaluation/benchmark. Use `generation.temperature`/`top_p`/`max_tokens` +and supported `proxy.extra_body` fields. Otherwise preserve checkpoint/vLLM +defaults, not general card recommendations or guessed caps. Inspect resolved +configs and requests: evaluator/agent defaults may populate omitted fields, +`null` is not omission, and interceptors may remove explicit caps. Ensure +required settings reach the server; do not silently strip them. ## One benchmark per config diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md index f1f91d80a60..0a249fdad83 100644 --- a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -9,56 +9,35 @@ those whose cards disclose nothing usable, and **speculative-decoding variants not change its output distribution, so those checkpoints share their base checkpoint's row. A miss here means "read the card", not "not yet checked". -Use it to reproduce a published NVFP4 number, and as the cross-check when a card -is silent or ambiguous. It does not replace reading the card — see -`model-card-research.md`. +Use this dated snapshot to cross-check published settings, not to fill gaps in +silent or ambiguous cards. Read the exact card and verify each field's evaluation +provenance — see `model-card-research.md`. ## Lookup -**The card is the source of truth; this table is a reference, not a constraint.** -Use it to confirm what you read, to fill a gap when the card is silent, and as a -sanity check when you are unsure — never to override a value the card states. - -1. **Exact row, resolved per field.** `eval` → use it, cite the row. `rec` → use - it, but note in the config comment that it is recommended sampling, not a - stated eval setting; if a same-family `eval` row disagrees, surface both. - `—` → that field is unpublished; resolve **it alone** via step 2. - `max_num_tokens` is the card's *headline* cap — where a note names a higher - per-task cap (GLM-5.2 GPQA `100000`, Qwen3.5-397B-V2 τ²-Telecom `128000`, - Kimi-K3 uncapped for Terminal-Bench) and that task is in your suite, SKILL.md - Step 3's take-the-highest rule governs the single top-level value, not the - column. -2. **No row** (new or unreleased variant, non-NVIDIA baseline, pre-2026) → take - the nearest same-family rows as the expected value. -3. **Card vs. table.** Agree → proceed. Card silent + family consistent → adopt - the family value and cite this file in a line comment; that beats SKILL.md - Step 3's generic 65536 / 16384. **Card disagrees → the card wins**, but - surface it — defaults shift between generations, so a mismatch means re-read, - not auto-correct. -4. **Baseline and candidate share one setting.** Cards report both precisions - measured under the single setting listed; use the NVFP4 row for both. - -**Per-task sampling is precedent, not mandate.** Some notes record a -benchmark-specific `temperature` / `top_p` (Qwen3.6 SciCode `0.6`; Qwen3.6-27B -τ²-Bench Telecom `0.0` / `top_p=1.0`; Kimi-K3 `top_p=1.0` agentic). Engineers do -tune sampling per benchmark, so **follow the card you are working from** and use -these as the cross-check. Where the two disagree, **escalate to the user on a -regime change, not a nudge** — greedy (`temperature ≤ 0.1` or `top_p ≤ 1e-4`) -versus sampled flips the regime and materially moves both score and variance; -`0.95` vs `1.0` does not. NEL accepts per-task `temperature` / `top_p` under -`evaluation.tasks.*.nemo_evaluator_config`; only `max_new_tokens` is barred -(SKILL.md Step 3). - -> **Never take sampling from a card's quickstart snippet.** -> `SamplingParams(temperature=0.8, top_p=0.95)` and `max_tokens=32` are -> boilerplate, repeated verbatim across unrelated models. Only *Benchmarked -> with…* / *evaluated with…* / *We evaluate the model using…* sentences, -> "Recommended Sampling" rows, and footnotes under the accuracy table count. - -`provenance` — **`eval`** (20 rows): card ties the values to its accuracy table, -authoritative. **`rec`** (5 rows): card recommends them for inference without -that tie. `max_num_tokens` is the max generation length, i.e. -`nemo_evaluator_config.config.params.max_new_tokens`. +**Explicit user/task requirements take precedence over model-card settings.** + +1. **Exact row, resolved per field.** `eval` → verify the card explicitly ties + that field to evaluation/benchmarking for the applicable task/mode, then cite + the statement. A row can mix evaluation settings and general recommendations + in its notes; the row label does not authorize every value. `rec` and `—` do + not justify overrides. +2. **Missing or ambiguous evidence** → preserve `config.json`, + `generation_config.json`, and vLLM defaults. No same-family inference or + generic fallback. Verify evaluator defaults and actual requests as described + in SKILL.md Step 3; omission and explicit `null` are not interchangeable. +3. **Card vs. table** → re-read the card and surface discrepancies; only verified + evaluation settings qualify. Quickstarts, supported limits, and general + "Recommended Sampling" rows do not qualify without an explicit evaluation tie. +4. **Scope overrides to the stated tasks/modes**, including output caps. Do not + promote a task-specific or headline maximum to a suite-wide value. Apply + verified settings consistently to baseline and candidate when reproducing + their published comparison; otherwise confirm effective config parity. + +`provenance` — **`eval`** (20 rows): recorded as tied to evaluation; verify each +field against the card. **`rec`** (5 rows): inference recommendations only, not +an override source. `max_num_tokens` records generation length, corresponding +to `nemo_evaluator_config.config.params.max_new_tokens`, not context length. | Model card ID | temp | top_p | max_num_tokens | prov | notes | | --- | --- | --- | --- | --- | --- | @@ -88,26 +67,6 @@ that tie. `max_num_tokens` is the max generation length, i.e. | `nvidia/NVIDIA-Nemotron-Labs-3-Elastic-30B-A3B-NVFP4` | 1.0 | 1.0 | — | rec | reasoning tasks | | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` | 0.6 think / 0.2 instruct | 0.95 think / — | 20480 think / 1024 instruct | rec | think adds `reasoning_budget=16384`, `grace_period=1024`; instruct sets `top_k=1` | -## Priors (verify against the card) - -- **`1.0 / 0.95` is the house default** — GLM 4.7–5.2, Kimi K2.6–K3, MiniMax - M2.5–M3, Gemma 4, Nemotron 3/3.5, Qwen3.6. Best guess when a recent card is - silent. -- **DeepSeek is carved out of it** — its row uses `top_p=1.0`, not `0.95`. - Never carry the house default onto an unlisted DeepSeek variant. -- **Qwen splits by variant** — thinking `0.6 / 0.95` at Qwen3.5, raised to `1.0` - at Qwen3.6; instruct/coder near-greedy `0 / 1e-5` with `16384`. -- **GLM** — `1.0 / 0.95` throughout; cap fell from 131072 (4.7, 5) to 64000 - (5.1, 5.2). -- **Caps cluster at 64000 / 65536 / 81920 / 128000 / 131072**, 64000 most - common. `16384` appears only with greedy instruct Qwen; DeepSeek-V4-Flash's - `384000` is a long-context outlier. -- **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench - Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench - (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`, so when a card - lists two caps **take the maximum** as the single top-level value and note the - split in a comment. - ## Refreshing Built from the HF API, verified to match the website pagination page for page diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 3a33a6af053..e2565210c3e 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -10,7 +10,8 @@ "Runs 'nel skills build-config' with correct flags matching user answers, using NEL's current model_type terminology for the chat/reasoning template family", "Searches the web for the model card on HuggingFace and extracts model-specific settings", "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", - "Sets reasoning sampling params from model card: temperature=1.0, top_p=1.0", + "Overrides generation params from the model card only when explicitly tied to evaluation/benchmarking for the applicable task/mode; otherwise preserves verified checkpoint/server defaults", + "Honors explicit user/task generation requirements ahead of model-card settings and cites the source of each override", "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)", "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false", "Adds deployment.pre_cmd using curl (not wget) to download nano_v3_reasoning_parser.py from HuggingFace", @@ -43,7 +44,8 @@ "Auto-detects quantization format by reading ./llama-3.1-8b-fp8/config.json first, then ./llama-3.1-8b-fp8/hf_quant_config.json if needed", "Finds quant_algo=FP8 and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", "Recommends accuracy-sensitive benchmarks from references/quantization-benchmarks.md", - "Searches web for Llama-3.1-8B model card and extracts sampling params, context length, TP settings", + "Reads the Llama-3.1-8B model card and checkpoint config.json/generation_config.json; distinguishes evaluation sampling from inference recommendations and context limits", + "Does not fill missing evaluation settings from same-family rows or generic 65536/16384 budgets; verifies effective evaluator requests rather than assuming omission or null preserves server defaults", "Fills in remaining missing values by asking user", "Runs dry-run, then a limited-samples canary with limit_samples=10, then full evaluation only after canary log checks pass", "Tunes parallelism from the canary and avoids over-raising it when benchmarks depend on inference judges or code execution containers", @@ -63,7 +65,8 @@ "Runs nel skills build-config with --execution slurm --deployment vllm --benchmarks standard", "Detects FP8 quantization from config.json or hf_quant_config.json and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", "Reads references/quantization-benchmarks.md and recommends accuracy-sensitive benchmarks", - "Uses WebSearch to research model card for sampling params and context length", + "Researches model-card evaluation provenance and context length; only applies generation overrides backed by explicit user/task requirements or applicable benchmark statements", + "Keeps benchmark-specific settings scoped to their tasks instead of choosing the highest value anywhere in the card", "Fills in SLURM-specific values: hostname, account, partition from user input", "Runs dry-run validation and a limited-samples canary before full evaluation", "Checks canary logs for judge, container, deployment, and code-execution failures before treating the run as ready", diff --git a/plugins/modelopt/skills/launching-evals/references/analyze-results.md b/plugins/modelopt/skills/launching-evals/references/analyze-results.md index 48e0a87bb59..82576e7f3df 100644 --- a/plugins/modelopt/skills/launching-evals/references/analyze-results.md +++ b/plugins/modelopt/skills/launching-evals/references/analyze-results.md @@ -43,11 +43,11 @@ Check logs for silent errors that may invalidate results: ## Step 3: Validate config and methodology 1. **Methodology consistency**: Verify same benchmark versions, prompt templates, sampling params, and infrastructure across all models. Flag discrepancies. -2. **HF model card compliance**: Read the model's HuggingFace model card. Flag any deviations in inference parameters (temperature, top_p, max_new_tokens, deployment args, reasoning flags, etc.). -3. **Reasoning model validation**: Verify temp > 0, top_p > 0, `max_tokens` = null (allow full output length). +2. **Generation settings**: Follow the evaluation skill's provenance policy: explicit user/task requirements first, then applicable model-card settings explicitly used for evaluation/benchmarking; otherwise preserve checkpoint/vLLM defaults. Flag unsupported overrides, not deviations from general inference recommendations. +3. **Reasoning model validation**: Verify the intended reasoning mode and effective generation settings; do not impose generic sampling or token budgets. NOTE: `use_reasoning: False` in adapter_config does NOT mean reasoning is disabled — it only controls the reasoning interceptor. Whether reasoning is active depends on the model's own controls (deployment args, system prompt, API payload fields, etc.). -4. **Non-reasoning model validation**: Verify `max_tokens` = 16k -5. **Max model length**: Verify `max-model-len` = 131072 (leaderboard-recommended). Long context benchmarks (AA LCR, RULER) and agentic benchmarks may require a longer `max-model-len`. +4. **Effective requests**: Check evaluator/adapter defaults and actual requests. Omitted fields and explicit `null` are not interchangeable; confirm required settings reach the server. +5. **Max model length**: Check prompt + output capacity, including multi-turn history, against the supported context window. Do not infer generation limits from `max_position_embeddings` or impose a generic context override. 6. **RULER tasks**: Check thinking disabled, walltime=4h, rope-scaling for Qwen models 7. **AA baseline comparison**: Compare results against Artificial Analysis published scores. Exact match not expected — flag significant deviations. 8. **Model baseline comparison**: For quantized runs, compare results against the matching baseline model run when available. The baseline may be unquantized or simply less quantized (for example, FP8 as the baseline for NVFP4). Use the same benchmark version, task config, serving args, token limits, dataset setup, and infrastructure before treating the delta as a quantization effect. From 033ca34f604fa878b33b7df3a6d329a5f1cd23b4 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Tue, 15 Sep 2026 22:39:50 +0000 Subject: [PATCH 2/6] docs: align NEL-next schema with generation provenance policy Signed-off-by: Chad Voegele --- plugins/modelopt/skills/evaluation/references/nel-next.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index a21a5403707..3fd4701d2ee 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -86,7 +86,7 @@ services: extra_args: [...] # raw vllm flags — everything EXCEPT parallelism/served-model-name/port (see "vLLM deployment" below) extra_env: {...} # VLLM_* backend env (e.g. NVFP4 MoE flags) container_mounts: [/.cache/vllm:/cache/vllm, ...] - generation: {temperature: 1.0, top_p: 0.95} + generation: {} # add overrides only per Step 3's generation provenance/precedence policy proxy: {request_timeout: 3600, extra_body: {...}, interceptors: [...]} # >= llm_kwargs.timeout node_pool: gpu benchmarks: # EXACTLY ONE entry — one benchmark per config (see "One benchmark per config") From 4dde4b099a489047123871fb177e68de1142dfa4 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Wed, 16 Sep 2026 16:03:15 +0000 Subject: [PATCH 3/6] docs: preserve Artificial Analysis evaluation settings Signed-off-by: Chad Voegele --- plugins/modelopt/skills/day0-release/SKILL.md | 6 ++- plugins/modelopt/skills/evaluation/SKILL.md | 11 ++++-- .../recipes/examples/example_eval.yaml | 12 +++--- .../recipes/examples/example_eval_next.yaml | 10 ++--- .../recipes/examples/gym/example_gdpval.yaml | 4 +- .../recipes/examples/gym/example_mrcr.yaml | 4 +- .../evaluation/references/aa-methodology.md | 37 ++++++++++++++++++ .../references/model-card-research.md | 5 ++- .../skills/evaluation/references/nel-next.md | 3 +- .../references/nvfp4-modelcard-sampling.md | 9 +++-- .../skills/evaluation/tests/evals.json | 39 ++++++++++++++++++- .../references/analyze-results.md | 8 ++-- 12 files changed, 115 insertions(+), 33 deletions(-) create mode 100644 plugins/modelopt/skills/evaluation/references/aa-methodology.md diff --git a/plugins/modelopt/skills/day0-release/SKILL.md b/plugins/modelopt/skills/day0-release/SKILL.md index 3076b6e122d..fbde83bb578 100644 --- a/plugins/modelopt/skills/day0-release/SKILL.md +++ b/plugins/modelopt/skills/day0-release/SKILL.md @@ -144,8 +144,10 @@ diff <(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \ Any other difference biases the comparison and invalidates the gate: a mismatched `parallelism` between the two sides was worth ~2 pp, enough to invert the sign of the delta. Follow the evaluation skill's generation-parameter policy: explicit -user/task requirements first; model-card overrides only for values explicitly -used in evaluation/benchmarking of the applicable task/mode. Otherwise preserve +user/task requirements first, including the scoped +[AA task policy and provenance exception](../evaluation/references/aa-methodology.md); +otherwise model-card overrides only for values explicitly used in +evaluation/benchmarking of the applicable task/mode. Otherwise preserve `config.json`, `generation_config.json`, and vLLM defaults, without family-based or generic fallbacks. Apply any benchmark-specific split to both sides. Verify effective sampling parity too: identical client configs can inherit different diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 1d7e1d3a089..1cce386fe2f 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -311,14 +311,17 @@ nemo_evaluator_config: params: parallelism: ??? # Required — size per references/parallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear request_timeout: 3600 - max_retries: 10 + max_retries: 10 # Operational default; AA reproduction requires attempt-semantics verification below ``` #### Generation parameters — provenance and precedence -1. **Explicit user/task requirements take precedence.** Preserve required token - budgets and sampling settings; do not replace them with model-card values. -2. **Read the full model card before deriving overrides.** Override +1. **Explicit user/task requirements take precedence.** For applicable AA + benchmark/version reproduction, apply the authoritative + [AA task policy](references/aa-methodology.md), including its deliberate + exceptions to model-card evaluation provenance. Preserve required budgets and + sampling settings; surface conflicts rather than claiming AA reproduction. +2. **Otherwise read the full model card before deriving overrides.** Override `max_new_tokens` / `max_tokens`, `temperature`, `top_p`, or other generation parameters only when the card explicitly says they were used for evaluation or benchmarking. Cite the statement and its applicable tasks/mode. General diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml index 5e115601c3d..3c66d518b3d 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml @@ -132,13 +132,11 @@ evaluation: params: parallelism: ??? # Number of concurrent requests per each benchmark request_timeout: 3600 - max_retries: 10 - # Add generation overrides for explicit user/task requirements first. - # Otherwise use model-card values ONLY when explicitly used for evaluation/ - # benchmarking of the applicable task/mode (cite the statement). - # Without that evidence, preserve config.json/generation_config.json/vLLM - # defaults. Omission can still activate evaluator defaults: inspect resolved - # configs and canary requests (SKILL Step 3). null is not omission. + max_retries: 10 # For AA reproduction, verify total attempts per references/aa-methodology.md. + # Apply SKILL Step 3, including references/aa-methodology.md for applicable + # AA reproduction and its provenance exception. Otherwise use evaluation- + # provenanced card values or checkpoint/vLLM defaults; inspect requests. + # null is not omission. Update export tags for any explicit overrides. target: api_endpoint: api_key_name: DUMMY_API_KEY diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 873e06f4aeb..fe2d15e5459 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -41,14 +41,14 @@ services: container_mounts: # source dirs MUST pre-exist (pyxis won't create them): ssh 'mkdir -p //.cache/{vllm,huggingface}' - ???:/cache/vllm - ???:/cache/huggingface - # Add generation fields for explicit user/task requirements first; otherwise - # only for model-card settings explicitly used for evaluation/benchmarking. - # Preserve checkpoint/vLLM defaults absent that evidence. Omission can still - # activate evaluator/agent defaults; inspect requests. null is not omission. + # Apply SKILL Step 3, including references/aa-methodology.md for applicable + # AA reproduction and its provenance exception. Otherwise use evaluation- + # provenanced card values or checkpoint/vLLM defaults; inspect requests. + # null is not omission. generation: {} # see SKILL Step 3; mirror explicit values in tags proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout - extra_body: {skip_special_tokens: false} # add only requirement- or evaluation-provenanced generation extras; mirror in tags + extra_body: {skip_special_tokens: false} # generation extras follow SKILL Step 3 (including scoped AA policy); mirror in tags interceptors: - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml index 73b0e991cc5..a0a945d8521 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml @@ -167,8 +167,8 @@ evaluation: nemo_evaluator_config: config: params: - # Resolve per SKILL Step 3: explicit user/task requirements, then applicable - # evaluation-provenanced card values, otherwise verified checkpoint/vLLM defaults. + # Resolve per SKILL Step 3, including references/aa-methodology.md only for + # applicable AA reproduction; otherwise retain the normal provenance/default policy. # These fields feed collect_rollout_params below; if omitting them, remove # those overrides too and verify Gym/evaluator defaults in actual requests. # Do not use null as a substitute for omission. Update export tags together. diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 986cc4e4e91..7df8cb44ecf 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -129,8 +129,8 @@ evaluation: nemo_evaluator_config: config: params: - # Resolve per SKILL Step 3: explicit user/task requirements, then applicable - # evaluation-provenanced card values, otherwise verified checkpoint/vLLM defaults. + # Resolve per SKILL Step 3, including references/aa-methodology.md only for + # applicable AA reproduction; otherwise retain the normal provenance/default policy. # These fields feed collect_rollout_params below; if omitting them, remove # those overrides too and verify Gym/evaluator defaults in actual requests. # Do not use null as a substitute for omission. Update export tags together. diff --git a/plugins/modelopt/skills/evaluation/references/aa-methodology.md b/plugins/modelopt/skills/evaluation/references/aa-methodology.md new file mode 100644 index 00000000000..e8b14ba9f01 --- /dev/null +++ b/plugins/modelopt/skills/evaluation/references/aa-methodology.md @@ -0,0 +1,37 @@ +# Artificial Analysis task policy + +When reproducing an applicable AA benchmark/version, treat the +[AA intelligence benchmarking methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking) +(General Testing Parameters and benchmark-specific sections) as authoritative +task policy, ahead of generic model-card evaluation-provenance/default rules. +Confirm applicability and record the benchmark/version and sources; an AA-style +prompt or a generic GPQA run alone does not establish AA reproduction. Do not +migrate benchmark versions implicitly. Surface conflicting user requirements as +deviations from AA reproduction. + +- **Temperature:** `0` for non-reasoning, `0.6` for reasoning, unless the model lab + recommends another temperature for the applicable model/mode. +- **Output:** non-reasoning `16384` tokens, adjusted downward for a smaller context + window or output cap. Reasoning uses the maximum output tokens allowed as + disclosed by the model creators, resolved separately for each reasoning model. + Do not substitute a generic `65536`, a context-window size, or universal `null`. +- **Deliberate provenance exception:** AA permits the lab-recommended temperature + and requires the disclosed reasoning output maximum without a sentence tying + those values to evaluation. Cite the exact model/mode disclosure, not a related + model or arbitrary quickstart. Missing or ambiguous disclosures require + clarification, not a guessed cap. +- **`top_p`:** no general AA setting; use the normal provenance/default policy + unless an applicable benchmark-specific requirement supplies one. +- **Context:** AA-LCR v1.1 requires a minimum 128K context, not a universal exact + `--max-model-len 131072`. Verify supported prompt + output capacity, including + accumulated history; report infeasible budgets rather than silently shrinking + required reasoning output. +- **API failures:** automatic retry up to **30 total attempts**. Verify the + harness's retry semantics and nested retry layers: if `max_retries` counts + retries after the initial attempt, 30 attempts means 29 retries, not 30. + Do not change benchmark repeat counts to implement retries. + +Apply overrides only to applicable tasks. Inspect resolved requests after +adapters/interceptors, keep baseline/candidate settings aligned, and update +export tags and provenance. Outside this scope, retain the normal +[generation-parameter policy](../SKILL.md#generation-parameters--provenance-and-precedence). diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index 2cd13e25dd0..ead1a9c33c6 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -3,8 +3,9 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it carefully, the FULL text, the devil is in the details. Extract ALL relevant configurations: - **Generation params (`temperature`, `top_p`, `max_new_tokens`, etc.).** - Explicit user/task requirements take precedence. Derive model-card overrides - only from statements tying each value to evaluation/benchmarking ("Benchmarked + Explicit user/task requirements, including the scoped + [AA task policy and provenance exception](aa-methodology.md), take precedence. + Otherwise derive model-card overrides only from statements tying each value to evaluation/benchmarking ("Benchmarked with…", "…were evaluated with…"). General "Recommended Sampling" rows, quickstarts, and supported output/context limits are not evaluation evidence. Cite the statement and scope each override to the stated benchmark/mode; diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 3fd4701d2ee..5d52d4c7534 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -133,7 +133,8 @@ SKILL.md Step 3 (same vLLM). The 0.2.6 `command:` maps to structured `services.< Size TP/DP + backend defaults (`--max-num-seqs = ceil(max_parallelism/DP)`, MoE `--enable-expert-parallel`, …) per `references/parallelism.md` + Step 3. **Generation overrides follow Step 3's provenance/precedence policy:** explicit -user/task requirements first, then only card values explicitly used for the +user/task requirements first, including the scoped [AA task policy](aa-methodology.md) +and its provenance exception; otherwise only card values explicitly used for the applicable evaluation/benchmark. Use `generation.temperature`/`top_p`/`max_tokens` and supported `proxy.extra_body` fields. Otherwise preserve checkpoint/vLLM defaults, not general card recommendations or guessed caps. Inspect resolved diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md index 0a249fdad83..ff5a9e089ac 100644 --- a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -10,12 +10,15 @@ not change its output distribution, so those checkpoints share their base checkpoint's row. A miss here means "read the card", not "not yet checked". Use this dated snapshot to cross-check published settings, not to fill gaps in -silent or ambiguous cards. Read the exact card and verify each field's evaluation +silent or ambiguous cards. Read the exact card and verify each field's required provenance — see `model-card-research.md`. ## Lookup **Explicit user/task requirements take precedence over model-card settings.** +For applicable AA reproduction, use the [AA task policy](aa-methodology.md), +including its lab-temperature/disclosed-output exception. The evaluation-only +rules below apply otherwise; `rec` alone never authorizes every field. 1. **Exact row, resolved per field.** `eval` → verify the card explicitly ties that field to evaluation/benchmarking for the applicable task/mode, then cite @@ -35,8 +38,8 @@ provenance — see `model-card-research.md`. their published comparison; otherwise confirm effective config parity. `provenance` — **`eval`** (20 rows): recorded as tied to evaluation; verify each -field against the card. **`rec`** (5 rows): inference recommendations only, not -an override source. `max_num_tokens` records generation length, corresponding +field against the card. **`rec`** (5 rows): inference recommendations, not +ordinary evaluation evidence. `max_num_tokens` records generation length, corresponding to `nemo_evaluator_config.config.params.max_new_tokens`, not context length. | Model card ID | temp | top_p | max_num_tokens | prov | notes | diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index e2565210c3e..202a6c4c51d 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -10,7 +10,7 @@ "Runs 'nel skills build-config' with correct flags matching user answers, using NEL's current model_type terminology for the chat/reasoning template family", "Searches the web for the model card on HuggingFace and extracts model-specific settings", "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", - "Overrides generation params from the model card only when explicitly tied to evaluation/benchmarking for the applicable task/mode; otherwise preserves verified checkpoint/server defaults", + "Applies authoritative task policy first (including the scoped AA exception if reproducing AA); otherwise overrides card params only with applicable evaluation provenance or preserves verified checkpoint/server defaults", "Honors explicit user/task generation requirements ahead of model-card settings and cites the source of each override", "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)", "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false", @@ -75,6 +75,43 @@ "Provides SSH-based log monitoring commands for SLURM execution" ] }, + { + "name": "aa-generation-policy", + "skills": ["evaluation"], + "query": "Prepare an AA reproduction config for a reasoning model on GPQA and AA-LCR v1.1, without submitting. The exact model lab recommends temperature 0.7 and discloses a maximum output of 98304 tokens, but neither statement mentions evaluation. It supports 256K context. There is no top_p guidance. Keep these benchmark versions.", + "files": [], + "expected_behavior": [ + "Reads the canonical AA methodology reference and treats applicable AA policy as authoritative ahead of generic evaluation-only card rules", + "Uses the lab-recommended temperature 0.7 and creator-disclosed reasoning output maximum 98304 without demanding an evaluation-provenance sentence; cites both disclosures", + "Explains that absent a lab temperature recommendation AA uses 0 for non-reasoning and 0.6 for reasoning", + "Does not invent top_p=0.95, a generic reasoning 65536 cap, or universal null; verifies effective requests and required budgets after adapters", + "Treats AA-LCR v1.1's 128K as a minimum, not an exact 131072 override; checks prompt plus output capacity and reports conflicts", + "Verifies retry semantics for up to 30 total API attempts including nested retries, rather than blindly setting max_retries=30", + "Scopes AA settings to applicable benchmark/version reproduction, preserves versions, and updates export provenance and tags" + ] + }, + { + "name": "aa-nonreasoning-output", + "skills": ["evaluation"], + "query": "Prepare non-reasoning AA GPQA reproduction configs for two models with no lab temperature recommendation. One supports 16384 output tokens; the other has a disclosed 8192 output cap. Both have ample context. Do not submit.", + "files": [], + "expected_behavior": [ + "Uses AA temperature 0 and output 16384 for the first model, reducing output to 8192 for the second with a cited capacity constraint", + "Does not require an evaluation-provenance sentence to apply AA methodology or spread the reduced cap to unrelated models/tasks", + "Does not impose a general AA top_p or exact 131072 context override; verifies effective requests" + ] + }, + { + "name": "non-aa-gpqa-defaults", + "skills": ["evaluation"], + "query": "Prepare a generic GPQA evaluation, not an AA reproduction. My reasoning model card has only general inference recommendations and supported output limits, with no evaluation settings. Do not submit.", + "files": [], + "expected_behavior": [ + "Does not infer AA reproduction from GPQA or its prompt format and does not apply the AA lab-recommendation exception", + "Preserves verified checkpoint/server defaults rather than imposing temperatures 0.6 or 1, top_p 0.95, output 16384 or 65536, a creator maximum, or null", + "Inspects evaluator defaults and effective requests; cites applicable evaluation evidence before any card-derived override" + ] + }, { "name": "scicode-eight-run-average", "skills": ["evaluation"], diff --git a/plugins/modelopt/skills/launching-evals/references/analyze-results.md b/plugins/modelopt/skills/launching-evals/references/analyze-results.md index 82576e7f3df..d77aa78a799 100644 --- a/plugins/modelopt/skills/launching-evals/references/analyze-results.md +++ b/plugins/modelopt/skills/launching-evals/references/analyze-results.md @@ -43,11 +43,11 @@ Check logs for silent errors that may invalidate results: ## Step 3: Validate config and methodology 1. **Methodology consistency**: Verify same benchmark versions, prompt templates, sampling params, and infrastructure across all models. Flag discrepancies. -2. **Generation settings**: Follow the evaluation skill's provenance policy: explicit user/task requirements first, then applicable model-card settings explicitly used for evaluation/benchmarking; otherwise preserve checkpoint/vLLM defaults. Flag unsupported overrides, not deviations from general inference recommendations. -3. **Reasoning model validation**: Verify the intended reasoning mode and effective generation settings; do not impose generic sampling or token budgets. +2. **Generation settings**: For applicable AA benchmark/version reproduction, validate against the [AA task policy](../../evaluation/references/aa-methodology.md), including its provenance exception. Otherwise follow the evaluation skill's normal user/task → evaluation-provenanced card → checkpoint/vLLM defaults policy. +3. **Reasoning model validation**: For AA reproduction, verify the creator-disclosed maximum output and AA temperature policy, not generic 64K or `null`. Otherwise verify effective settings without imposing generic sampling or budgets. Verify the intended reasoning mode. NOTE: `use_reasoning: False` in adapter_config does NOT mean reasoning is disabled — it only controls the reasoning interceptor. Whether reasoning is active depends on the model's own controls (deployment args, system prompt, API payload fields, etc.). -4. **Effective requests**: Check evaluator/adapter defaults and actual requests. Omitted fields and explicit `null` are not interchangeable; confirm required settings reach the server. -5. **Max model length**: Check prompt + output capacity, including multi-turn history, against the supported context window. Do not infer generation limits from `max_position_embeddings` or impose a generic context override. +4. **Non-reasoning and effective requests**: For AA reproduction, verify `max_tokens` = `16384`, reduced only for smaller context/output limits per the AA policy. Otherwise preserve the normal default policy. Check evaluator/adapter defaults and actual requests; omission and `null` are not interchangeable. +5. **Max model length**: Check prompt + output capacity, including multi-turn history, against the supported context window and applicable task minimum (AA-LCR v1.1: 128K). Do not infer generation limits from `max_position_embeddings` or require exactly 131072 for every run. 6. **RULER tasks**: Check thinking disabled, walltime=4h, rope-scaling for Qwen models 7. **AA baseline comparison**: Compare results against Artificial Analysis published scores. Exact match not expected — flag significant deviations. 8. **Model baseline comparison**: For quantized runs, compare results against the matching baseline model run when available. The baseline may be unquantized or simply less quantized (for example, FP8 as the baseline for NVFP4). Use the same benchmark version, task config, serving args, token limits, dataset setup, and infrastructure before treating the delta as a quantization effect. From 37fe654ff29b645f77a549d28d21196a640be428 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Thu, 17 Sep 2026 18:11:41 +0000 Subject: [PATCH 4/6] docs: narrow evaluation guidance fix to output token budgets Signed-off-by: Chad Voegele --- plugins/modelopt/skills/day0-release/SKILL.md | 12 +-- plugins/modelopt/skills/evaluation/SKILL.md | 71 +++++--------- .../recipes/examples/example_eval.yaml | 30 +++--- .../recipes/examples/example_eval_next.yaml | 15 +-- .../recipes/examples/gym/example_gdpval.yaml | 15 +-- .../recipes/examples/gym/example_mrcr.yaml | 15 +-- .../tasks/aa_next/swebench_verified.md | 2 +- .../evaluation/references/aa-methodology.md | 37 -------- .../evaluation/references/gym-gdpval.md | 9 +- .../references/model-card-research.md | 45 +++++---- .../skills/evaluation/references/nel-next.md | 16 +--- .../references/nvfp4-modelcard-sampling.md | 95 +++++++++++++------ .../skills/evaluation/tests/evals.json | 46 +-------- .../references/analyze-results.md | 8 +- 14 files changed, 162 insertions(+), 254 deletions(-) delete mode 100644 plugins/modelopt/skills/evaluation/references/aa-methodology.md diff --git a/plugins/modelopt/skills/day0-release/SKILL.md b/plugins/modelopt/skills/day0-release/SKILL.md index fbde83bb578..ee58d078219 100644 --- a/plugins/modelopt/skills/day0-release/SKILL.md +++ b/plugins/modelopt/skills/day0-release/SKILL.md @@ -143,16 +143,8 @@ diff <(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \ Any other difference biases the comparison and invalidates the gate: a mismatched `parallelism` between the two sides was worth ~2 pp, enough to invert the sign of -the delta. Follow the evaluation skill's generation-parameter policy: explicit -user/task requirements first, including the scoped -[AA task policy and provenance exception](../evaluation/references/aa-methodology.md); -otherwise model-card overrides only for values explicitly used in -evaluation/benchmarking of the applicable task/mode. Otherwise preserve -`config.json`, `generation_config.json`, and vLLM defaults, without family-based -or generic fallbacks. Apply any benchmark-specific split to both sides. Verify -effective sampling parity too: identical client configs can inherit different -checkpoint or evaluator defaults; omission is not explicit `null`. -Never set an unbounded `request_timeout` (`1e9`) — it turns a +the delta. If the model card splits sampling params per scenario, apply the same +split to both sides. Never set an unbounded `request_timeout` (`1e9`) — it turns a transient stall into a job that holds its GPUs until the wall clock kills it. Gate: diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 1cce386fe2f..9fb70bc2857 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -144,7 +144,7 @@ Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. 1. Read the task reference file(s). 2. Use `recipes/examples/example_eval.yaml` as the base. 3. Copy the YAML fragment(s) into `evaluation.tasks`, applying any per-task notes. -4. **MLflow auto-export is on by default** — it needs **two** pieces, both in `example_eval.yaml`: (a) the **trigger** `execution.auto_export.destinations: [mlflow]` (without it the run is *not* uploaded), and (b) the `export.mlflow` block that configures it. In the `export.mlflow` block use **literal** values for `experiment_name` / `description` / `tags` — substitute the actual `served_model_name` and sampling params. Do **not** use `${deployment.*}` / `${evaluation.*}` cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with `Interpolation key '...' not found` (`${oc.env:USER}` and `${oc.env:MLFLOW_TRACKING_URI}` are fine — they're env vars). Because these literals can't interpolate, keep the `temperature` / `top_p` / `max_new_tokens` tags **equal to** explicit top-level `params` (label absent fields `omitted`, not an assumed effective value) and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. `tracking_uri` = `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config` (not hand-filled), and auto-export needs `execution.cpu_partition` (e.g. gcp-nrt `cpu`) — it's a separate CPU-only sbatch that GPU-only partitions reject (`Cannot find GPU specification`), silently dropping the link. Before filling `experiment_name`/`tags`, read the checkpoint's `.experiment.json` (Step 3) and carry the PTQ run's experiment name plus its `modelopt_*` tags across (Step 4). +4. **MLflow auto-export is on by default** — it needs **two** pieces, both in `example_eval.yaml`: (a) the **trigger** `execution.auto_export.destinations: [mlflow]` (without it the run is *not* uploaded), and (b) the `export.mlflow` block that configures it. In the `export.mlflow` block use **literal** values for `experiment_name` / `description` / `tags` — substitute the actual `served_model_name` and sampling params. Do **not** use `${deployment.*}` / `${evaluation.*}` cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with `Interpolation key '...' not found` (`${oc.env:USER}` and `${oc.env:MLFLOW_TRACKING_URI}` are fine — they're env vars). Because these literals can't interpolate, keep the `temperature` / `top_p` / `max_new_tokens` tags **equal to** the top-level `params` and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. `tracking_uri` = `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config` (not hand-filled), and auto-export needs `execution.cpu_partition` (e.g. gcp-nrt `cpu`) — it's a separate CPU-only sbatch that GPU-only partitions reject (`Cannot find GPU specification`), silently dropping the link. Before filling `experiment_name`/`tags`, read the checkpoint's `.experiment.json` (Step 3) and carry the PTQ run's experiment name plus its `modelopt_*` tags across (Step 4). 5. Proceed to Step 3, then Step 4, then Step 7.5/8. Skip Step 2's 5-question flow. --- @@ -266,7 +266,7 @@ deployment: <... rest of cross-checked flags ...> ``` -Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `--served-model-name ${deployment.served_model_name}` (**required**; see `example_eval.yaml` for why); always `--host 0.0.0.0 --port ${deployment.port}`; use folded scalar (`>-`) for one flag per line. Preserve the checkpoint/vLLM context default unless an explicit task requirement or verified deployment recipe requires an override; check that the model supports the requested context. +Conventions: always start `vllm serve /checkpoint` (NEL mounts here); always `--served-model-name ${deployment.served_model_name}` (**required**; see `example_eval.yaml` for why); always `--host 0.0.0.0 --port ${deployment.port}`; use folded scalar (`>-`) for one flag per line. Example fallback `--max-model-len 131072` covers AA-LCR (~120K + 16K gen) and SciCode (≥ 65536) — prefer `config.json` / recipe value. For how to choose `--tensor-parallel-size` / `--data-parallel-size` / `--pipeline-parallel-size` (and EP) from the model size and your GPU count, read `references/parallelism.md` — cross-check the layout against `recipes.vllm.ai`, then adapt to the GPUs you actually have via the fit math there. @@ -300,10 +300,7 @@ Silence is not contradiction. Drop/override only when the recipe sets a differen #### Evaluation params template (top-level params) -Start with these operational fields in `nemo_evaluator_config.config.params`. -Add generation overrides only under the policy below; route `top_k` / -`presence_penalty` / `repetition_penalty` / `min_p` through the harness-supported -request adapter, not unsupported top-level fields. +The top-level `nemo_evaluator_config.config.params` must contain **exactly these six fields** — no `top_k` / `presence_penalty` / `repetition_penalty` / `min_p`: ```yaml nemo_evaluator_config: @@ -311,48 +308,30 @@ nemo_evaluator_config: params: parallelism: ??? # Required — size per references/parallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear request_timeout: 3600 - max_retries: 10 # Operational default; AA reproduction requires attempt-semantics verification below + max_retries: 10 + max_new_tokens: ??? # resolve from the token-budget rule below + temperature: 1.0 # from model card (reasoning); adjust + top_p: 0.95 # from model card (reasoning); adjust ``` -#### Generation parameters — provenance and precedence - -1. **Explicit user/task requirements take precedence.** For applicable AA - benchmark/version reproduction, apply the authoritative - [AA task policy](references/aa-methodology.md), including its deliberate - exceptions to model-card evaluation provenance. Preserve required budgets and - sampling settings; surface conflicts rather than claiming AA reproduction. -2. **Otherwise read the full model card before deriving overrides.** Override - `max_new_tokens` / `max_tokens`, `temperature`, `top_p`, or other generation - parameters only when the card explicitly says they were used for evaluation - or benchmarking. Cite the statement and its applicable tasks/mode. General - inference recommendations, quickstarts, supported limits, and unrelated - scenarios do not qualify. -3. **Otherwise preserve checkpoint/server defaults** from `config.json`, - `generation_config.json`, and vLLM. Do not invent reasoning/non-reasoning - fallbacks or borrow values from related models. Cross-check - `references/nvfp4-modelcard-sampling.md` against the exact card; never use it - to fill silent or ambiguous fields. -4. **Verify the effective request.** Omitting a client parameter does not bypass - evaluator/task defaults; inspect the resolved config and canary requests. - Where supported, remove unintended client overrides so server defaults apply, - or explicitly carry verified checkpoint/server values. An explicit `null` - is not omission or a universal "uncapped" setting: verify its meaning in the - selected harness/server. Report unresolved defaults rather than guessing. -5. **Keep settings scoped to their evidence.** Use shared top-level values only - where applicable; use per-task overrides (including token caps) for explicit - requirements or benchmark-specific card settings. Never take the highest - value anywhere in a card as a suite-wide cap. Apply the same policy to - baseline and candidate, and verify their effective settings match. - -Record explicit values and their sources in the config. Keep MLflow tags in -sync; label omitted client fields as `omitted`, not as assumed numeric defaults, -and record verified effective values and per-task differences in the description. - -For output length, distinguish generation limits from the server's context -window. Check prompt + requested output against `max_model_len`, including -accumulated history on multi-turn tasks. Surface conflicts instead of silently -shrinking required budgets. Higher caps do not fix runaway reasoning; inspect -`finish_reason: length` per `references/run-validation.md`. +Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere. + +**For `max_new_tokens`, the token-budget rule below takes precedence over table recommendations and family fallbacks.** + +**Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` / `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's "Lookup" section. + +**`temperature` / `top_p` are different: per-task overrides ARE allowed and often required.** Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives `top_p = 0.95` for agentic scenarios and `1.0` otherwise, so a single top-level `0.95` is wrong for every non-agentic task. +Set the top-level value for the majority case, override only the tasks the card calls out, and apply +the split identically to baseline and candidate. **The `export.mlflow` tags record only the +top-level values**, so note any per-task override in the run `description` — otherwise the +overridden task is reported under sampling params it did not use. + +#### `max_new_tokens` — mandatory model-card lookup + +1. **Read the HF model card before setting the value.** Identify the model's reasoning mode, creator-disclosed maximum output length, and any `max_tokens` / `max_new_tokens` used for the applicable evaluation. Cite the source and rationale in a config comment; quickstart examples are not evaluation budgets. +2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum; ask the user if it remains unknown. +3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs or user confirmation. +4. Verify that the output budget fits alongside the input, including accumulated multi-turn history. Surface conflicts rather than silently clipping a creator/evaluation budget. Check `finish_reason: length` after the run (`references/run-validation.md`); a higher cap does not fix runaway reasoning. #### Quantization-aware benchmark defaults diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml index 3c66d518b3d..41b7c4c8c40 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml @@ -99,8 +99,6 @@ deployment: # model as `/checkpoint` and eval requests 404 ("model does not exist"). # For MoE models, add `--enable-expert-parallel` to the command. # For models with custom code, add `--trust-remote-code` to the command. - # Preserve checkpoint/vLLM context defaults unless a task or verified deployment - # recipe requires --max-model-len; confirm the model supports that context. # `--model-loader-extra-config` enables multithreaded checkpoint loading (on by # default below) — cuts large-checkpoint load from tens of minutes to a few; # tune num_threads to the checkpoint size / shared-FS read bandwidth. @@ -114,6 +112,7 @@ deployment: --port ${deployment.port} --tensor-parallel-size 1 --data-parallel-size 1 + --max-model-len 131072 --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 128}' --max-num-batched-tokens 8192 --enable-chunked-prefill @@ -132,11 +131,10 @@ evaluation: params: parallelism: ??? # Number of concurrent requests per each benchmark request_timeout: 3600 - max_retries: 10 # For AA reproduction, verify total attempts per references/aa-methodology.md. - # Apply SKILL Step 3, including references/aa-methodology.md for applicable - # AA reproduction and its provenance exception. Otherwise use evaluation- - # provenanced card values or checkpoint/vLLM defaults; inspect requests. - # null is not omission. Update export tags for any explicit overrides. + max_retries: 10 + max_new_tokens: ??? # Resolve per SKILL.md's token-budget rule; match the export description/tag below + temperature: 1.0 # from model card (reasoning mode); adjust per card + top_p: 0.95 # from model card (reasoning mode); adjust per card target: api_endpoint: api_key_name: DUMMY_API_KEY @@ -164,8 +162,7 @@ evaluation: thinking: true enable_thinking: true # DeepSeek V4 only: reasoning-effort axis (high|max). "max" needs - # --max-model-len >= 393216. Uncomment only for an explicit requirement - # or an applicable, evaluation-provenanced model-card setting. + # --max-model-len >= 393216. Uncomment for DeepSeek V4. # reasoning_effort: max tasks: # Reasoning (chat endpoint, short). nemo-skills GPQA on the AA 4-choice MCQ @@ -190,10 +187,9 @@ export: # CAUTION — these literals can drift. temperature / top_p / max_new_tokens are the # ONLY queryable record of the sampling config in MLflow (NEL does not log them as # run params), so keep them — but because they can't be interpolated, they MUST be - # kept EQUAL to explicit config.params values above. 'omitted' means no client - # override here, NOT a verified server default; record effective values and - # per-task differences in the description after checking resolved requests. - # Update params and tags in the SAME edit, or MLflow will misreport the run. + # kept EQUAL to evaluation.nemo_evaluator_config.config.params above. When you + # change the sampling params (or served_model_name), update these literals in the + # SAME edit, or MLflow will misreport the run. mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config (canonical frontier-evals host; NOT the -nemo-evaluator alias) # If the checkpoint holds a .experiment.json, use ITS experiment_name verbatim here so @@ -201,7 +197,7 @@ export: # would resolve. Cross-server, that makes a same-named experiment here and only # modelopt_run_url reaches the PTQ run (Step 4). experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | generation overrides omitted; verify effective settings' + description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=???' log_logs: true log_artifacts: true only_required: false @@ -209,9 +205,9 @@ export: tags: framework: vllm model: CHANGEME-served-model-name - temperature: 'omitted' - top_p: 'omitted' - max_new_tokens: 'omitted' + temperature: '1.0' + top_p: '0.95' + max_new_tokens: ??? # From the checkpoint's .experiment.json when present; drop when absent. Keep the # quotes (a bare 20260910 becomes a date) and skip any value containing ${ -- quoting # does not stop OmegaConf resolving it. diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index fe2d15e5459..9463c653062 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -41,14 +41,10 @@ services: container_mounts: # source dirs MUST pre-exist (pyxis won't create them): ssh 'mkdir -p //.cache/{vllm,huggingface}' - ???:/cache/vllm - ???:/cache/huggingface - # Apply SKILL Step 3, including references/aa-methodology.md for applicable - # AA reproduction and its provenance exception. Otherwise use evaluation- - # provenanced card values or checkpoint/vLLM defaults; inspect requests. - # null is not omission. - generation: {} # see SKILL Step 3; mirror explicit values in tags + generation: {temperature: 1.0, top_p: 0.95} # from model card (reasoning mode); adjust per card — mandatory lookup (references/model-card-research.md), same as 0.2.6 proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout - extra_body: {skip_special_tokens: false} # generation extras follow SKILL Step 3 (including scoped AA policy); mirror in tags + extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below interceptors: - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped @@ -112,7 +108,6 @@ output: log_config_params: true copy_logs: true exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir - description: ??? # ' | verified effective sampling + sources | (timeout_strategy=…) | r8' - # model/checkpoint_path/benchmark drive dashboard attribution. Sampling tags - # mirror explicit generation fields; 'omitted' does NOT assert effective defaults. - tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: 'omitted', top_p: 'omitted'} + description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' + # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. + tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml index a0a945d8521..6e5eef250fa 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_gdpval.yaml @@ -167,13 +167,8 @@ evaluation: nemo_evaluator_config: config: params: - # Resolve per SKILL Step 3, including references/aa-methodology.md only for - # applicable AA reproduction; otherwise retain the normal provenance/default policy. - # These fields feed collect_rollout_params below; if omitting them, remove - # those overrides too and verify Gym/evaluator defaults in actual requests. - # Do not use null as a substitute for omission. Update export tags together. - temperature: ??? - top_p: ??? + temperature: 1.0 + top_p: 0.95 parallelism: 16384 # gym-internal concurrency, NOT a model-server cap request_timeout: 36000 # gym rollouts are long-running max_retries: 10 @@ -362,12 +357,12 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | GDPVal rubric | T=CHANGEME, top_p=CHANGEME, num_repeats=1' + description: 'CHANGEME-served-model-name | GDPVal rubric | T=1.0, top_p=0.95, num_repeats=1' log_logs: true only_required: false tags: framework: vllm model: CHANGEME-served-model-name benchmark: nemo_gym.gdpval - temperature: ??? - top_p: ??? + temperature: '1.0' + top_p: '0.95' diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 7df8cb44ecf..01c1c9a42c4 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -129,13 +129,8 @@ evaluation: nemo_evaluator_config: config: params: - # Resolve per SKILL Step 3, including references/aa-methodology.md only for - # applicable AA reproduction; otherwise retain the normal provenance/default policy. - # These fields feed collect_rollout_params below; if omitting them, remove - # those overrides too and verify Gym/evaluator defaults in actual requests. - # Do not use null as a substitute for omission. Update export tags together. - temperature: ??? - top_p: ??? + temperature: 1.0 + top_p: 0.95 # Gym client concurrency, TOTAL across instances — not a per-server cap. parallelism: 256 request_timeout: 36000 # 1M-token prefills are slow @@ -313,12 +308,12 @@ export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # from modelopttools:eval-config experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | MRCR 1M | T=CHANGEME, top_p=CHANGEME, num_repeats=1' + description: 'CHANGEME-served-model-name | MRCR 1M | T=1.0, top_p=0.95, num_repeats=1' log_logs: true only_required: false tags: framework: vllm model: CHANGEME-served-model-name benchmark: nemo_gym.mrcr_n3_1m # CHANGEME with the variant (see VARIANT SELECTOR) - temperature: ??? - top_p: ??? + temperature: '1.0' + top_p: '0.95' diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index e9095502ec2..bbd70551cb9 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -96,7 +96,7 @@ from the scored config. ```yaml proxy: request_timeout: 3600 - extra_body: {skip_special_tokens: false} # generation extras follow SKILL Step 3's provenance/precedence policy + extra_body: {skip_special_tokens: false} # add model-card sampling extras if the card sets them model_traffic: {capture_request_body: true} # FEA-224; adds the upstream request body to the traffic capture that is ALREADY ON by default interceptors: # - {name: http_pairs_dump, config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}} # canary only diff --git a/plugins/modelopt/skills/evaluation/references/aa-methodology.md b/plugins/modelopt/skills/evaluation/references/aa-methodology.md deleted file mode 100644 index e8b14ba9f01..00000000000 --- a/plugins/modelopt/skills/evaluation/references/aa-methodology.md +++ /dev/null @@ -1,37 +0,0 @@ -# Artificial Analysis task policy - -When reproducing an applicable AA benchmark/version, treat the -[AA intelligence benchmarking methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking) -(General Testing Parameters and benchmark-specific sections) as authoritative -task policy, ahead of generic model-card evaluation-provenance/default rules. -Confirm applicability and record the benchmark/version and sources; an AA-style -prompt or a generic GPQA run alone does not establish AA reproduction. Do not -migrate benchmark versions implicitly. Surface conflicting user requirements as -deviations from AA reproduction. - -- **Temperature:** `0` for non-reasoning, `0.6` for reasoning, unless the model lab - recommends another temperature for the applicable model/mode. -- **Output:** non-reasoning `16384` tokens, adjusted downward for a smaller context - window or output cap. Reasoning uses the maximum output tokens allowed as - disclosed by the model creators, resolved separately for each reasoning model. - Do not substitute a generic `65536`, a context-window size, or universal `null`. -- **Deliberate provenance exception:** AA permits the lab-recommended temperature - and requires the disclosed reasoning output maximum without a sentence tying - those values to evaluation. Cite the exact model/mode disclosure, not a related - model or arbitrary quickstart. Missing or ambiguous disclosures require - clarification, not a guessed cap. -- **`top_p`:** no general AA setting; use the normal provenance/default policy - unless an applicable benchmark-specific requirement supplies one. -- **Context:** AA-LCR v1.1 requires a minimum 128K context, not a universal exact - `--max-model-len 131072`. Verify supported prompt + output capacity, including - accumulated history; report infeasible budgets rather than silently shrinking - required reasoning output. -- **API failures:** automatic retry up to **30 total attempts**. Verify the - harness's retry semantics and nested retry layers: if `max_retries` counts - retries after the initial attempt, 30 attempts means 29 retries, not 30. - Do not change benchmark repeat counts to implement retries. - -Apply overrides only to applicable tasks. Inspect resolved requests after -adapters/interceptors, keep baseline/candidate settings aligned, and update -export tags and provenance. Outside this scope, retain the normal -[generation-parameter policy](../SKILL.md#generation-parameters--provenance-and-precedence). diff --git a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md index fd573d2abb8..f935e50ef57 100644 --- a/plugins/modelopt/skills/evaluation/references/gym-gdpval.md +++ b/plugins/modelopt/skills/evaluation/references/gym-gdpval.md @@ -143,11 +143,10 @@ self-deploys single-node vLLM, which is fine for a canary or a small policy. For SKILL Step 3/4's `ceil(parallelism / DP)` rule assumes `parallelism` is the in-flight request count; on the gym path it is not, and applying it literally gives an absurd cap. Use `ceil(stirrup_agent.concurrency / DP)` — e.g. 220 / DP 4 → 55, round to 64. -+ **`max_new_tokens`:** the template omits it and the adapter removes - `max_tokens` / `max_completion_tokens`. Follow Step 3's generation provenance - policy, not a golden config's incidental value. Omission can activate evaluator - defaults; inspect requests after adapters. If a user/task requires a cap, - ensure it reaches the server rather than being stripped. ++ **`max_new_tokens`:** the reviewed golden **does** set it alongside the adapter's + `params_to_remove: [max_tokens, max_completion_tokens]`, so do the Step 3 model-card + lookup as normal. The template omits it (five params) because the adapter strips the + per-request cap anyway; adding it back matches the golden and is harmless. + **Match `temperature` / `top_p` to whatever the reference deliverables were generated with.** A pairwise ELO compares your deliverables against theirs, so a sampling difference lands in the score as if it were a quality difference. diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index ead1a9c33c6..5d559cfe4f5 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -2,33 +2,36 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it carefully, the FULL text, the devil is in the details. Extract ALL relevant configurations: -- **Generation params (`temperature`, `top_p`, `max_new_tokens`, etc.).** - Explicit user/task requirements, including the scoped - [AA task policy and provenance exception](aa-methodology.md), take precedence. - Otherwise derive model-card overrides only from statements tying each value to evaluation/benchmarking ("Benchmarked - with…", "…were evaluated with…"). General "Recommended Sampling" rows, - quickstarts, and supported output/context limits are not evaluation evidence. - Cite the statement and scope each override to the stated benchmark/mode; - never choose the highest value across unrelated scenarios. - - Cross-check `nvfp4-modelcard-sampling.md` against the exact card, per field. - Do not fill gaps from the table or same-family models. - - If evidence is absent or ambiguous, preserve `config.json`, - `generation_config.json`, and vLLM defaults; no generic token-budget fallback. - Omitting client fields can still activate evaluator/task defaults. Inspect - resolved configs and canary requests, and remove unintended overrides or - carry verified checkpoint/server values through the supported harness path. - Do not substitute `null` for omission without verifying its semantics. - See SKILL.md Step 3 "Generation parameters — provenance and precedence". -- Context length (vLLM `--max-model-len`): preserve the checkpoint/server default - unless the task or verified deployment recipe requires an override; check - prompt + output capacity separately from generation limits. +- Sampling params (`temperature`, `top_p`) + - **Only trust a sentence that ties the values to the benchmarks** ("Benchmarked + with…", "…were evaluated with…", "We evaluate the model using…") or a + "Recommended Sampling" row. The `SamplingParams(temperature=0.8, top_p=0.95)` + in a card's TensorRT-LLM/vLLM quickstart snippet is boilerplate copied + verbatim across unrelated models — **never** read eval settings out of it. + - **Then cross-check `nvfp4-modelcard-sampling.md`** — published + `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints + under `huggingface.co/nvidia` that disclose them, grouped by family. + Required for any NVFP4 checkpoint or same-family sibling; it is also the + best source of a sampling default when the card is silent; output budgets + follow the rule below instead. Pre-2026 releases are out + of scope there — read their cards. +- Context length (`deployment.extra_args: "--max-model-len "`) +- **Output length (`max_new_tokens`) — mandatory extraction.** Record the + creator-disclosed maximum output length and any budget used for the applicable + evaluation, with sources. Non-reasoning defaults to **16384**, lowered for + smaller context/output caps; reasoning uses the creator-disclosed output + maximum (ask if unknown). An explicit model-card budget for the applicable + evaluation can override these defaults. No highest-number or same-family + fallback. Keep one top-level value; conflicting benchmark budgets require + separate configs or user confirmation. See SKILL.md Step 3 + "`max_new_tokens` — mandatory model-card lookup" for the full rule. - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed) - Reasoning config (if applicable): - reasoning on/off: use either: - `adapter_config.custom_system_prompt` (like `/think`, `/no_think`) and no `adapter_config.params_to_add` (leave `params_to_add` unrelated to reasoning untouched) - `adapter_config.params_to_add` for payload modifier (like `"chat_template_kwargs": {"enable_thinking": true/false}`) and no `adapter_config.custom_system_prompt` and `adapter_config.use_system_prompt: false` (leave `custom_system_prompt` and `use_system_prompt` unrelated to reasoning untouched). - **The `chat_template_kwargs` toggle key drifts across model generations — read the card / `chat_template.jinja`, don't extrapolate, and set only the one key the model uses.** Known: `enable_thinking` (Qwen3.5/3.6, GLM 5.1 — note GLM-4.x used `thinking`+`/nothink`); `thinking` (Kimi K2.6 — renamed from K2.5's `enable_thinking`; DeepSeek V3.2/V4 — Python encoder, not Jinja, so an unused kwarg can error rather than be ignored). - - reasoning effort/budget: apply the generation-parameter provenance policy above, not the highest documented effort by default. Honor tied deployment requirements (e.g. DeepSeek V4 Think Max needs `--max-model-len >= 393216`). + - reasoning effort/budget (if configurable, e.g. DeepSeek V4 `reasoning_effort`): **default to `max`** (the highest effort the card documents), honoring any tied requirement (e.g. V4 Think Max needs `--max-model-len >= 393216`). AskUserQuestion only if the user signals a cost/latency preference. - etc. - Deployment-specific `extra_args` for vLLM/SGLang (look for the vLLM/SGLang deployment command) - Deployment-specific vLLM/SGLang versions (by default we use latest docker images, but you can control it with `deployment.image` e.g. vLLM above `vllm/vllm-openai:v0.11.0` stopped supporting `rope-scaling` arg used by Qwen models) diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 5d52d4c7534..8e009098d9f 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -86,7 +86,7 @@ services: extra_args: [...] # raw vllm flags — everything EXCEPT parallelism/served-model-name/port (see "vLLM deployment" below) extra_env: {...} # VLLM_* backend env (e.g. NVFP4 MoE flags) container_mounts: [/.cache/vllm:/cache/vllm, ...] - generation: {} # add overrides only per Step 3's generation provenance/precedence policy + generation: {temperature: 1.0, top_p: 0.95} proxy: {request_timeout: 3600, extra_body: {...}, interceptors: [...]} # >= llm_kwargs.timeout node_pool: gpu benchmarks: # EXACTLY ONE entry — one benchmark per config (see "One benchmark per config") @@ -131,16 +131,10 @@ SKILL.md Step 3 (same vLLM). The 0.2.6 `command:` maps to structured `services.< | `image:` (bump to the model's recipe min; NVFP4 on sm_103 → CUDA-13 build, see Step 3) | `image:` (serving image, ≠ `eval_image`) | Size TP/DP + backend defaults (`--max-num-seqs = ceil(max_parallelism/DP)`, MoE -`--enable-expert-parallel`, …) per `references/parallelism.md` + Step 3. -**Generation overrides follow Step 3's provenance/precedence policy:** explicit -user/task requirements first, including the scoped [AA task policy](aa-methodology.md) -and its provenance exception; otherwise only card values explicitly used for the -applicable evaluation/benchmark. Use `generation.temperature`/`top_p`/`max_tokens` -and supported `proxy.extra_body` fields. Otherwise preserve checkpoint/vLLM -defaults, not general card recommendations or guessed caps. Inspect resolved -configs and requests: evaluator/agent defaults may populate omitted fields, -`null` is not omission, and interceptors may remove explicit caps. Ensure -required settings reach the server; do not silently strip them. +`--enable-expert-parallel`, …) per `references/parallelism.md` + Step 3. **Sampling +is a mandatory model-card lookup** (Step 3 / `references/model-card-research.md`, +never generic defaults): `generation.temperature`/`top_p` (+`max_tokens` if the card +caps output) and `proxy.extra_body` for any card extras (`skip_special_tokens`, thinking toggles). ## One benchmark per config diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md index ff5a9e089ac..a47d9f12a0f 100644 --- a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -9,38 +9,55 @@ those whose cards disclose nothing usable, and **speculative-decoding variants not change its output distribution, so those checkpoints share their base checkpoint's row. A miss here means "read the card", not "not yet checked". -Use this dated snapshot to cross-check published settings, not to fill gaps in -silent or ambiguous cards. Read the exact card and verify each field's required -provenance — see `model-card-research.md`. +Use it to reproduce a published NVFP4 number, and as the cross-check when a card +is silent or ambiguous. It does not replace reading the card — see +`model-card-research.md`. ## Lookup -**Explicit user/task requirements take precedence over model-card settings.** -For applicable AA reproduction, use the [AA task policy](aa-methodology.md), -including its lab-temperature/disclosed-output exception. The evaluation-only -rules below apply otherwise; `rec` alone never authorizes every field. - -1. **Exact row, resolved per field.** `eval` → verify the card explicitly ties - that field to evaluation/benchmarking for the applicable task/mode, then cite - the statement. A row can mix evaluation settings and general recommendations - in its notes; the row label does not authorize every value. `rec` and `—` do - not justify overrides. -2. **Missing or ambiguous evidence** → preserve `config.json`, - `generation_config.json`, and vLLM defaults. No same-family inference or - generic fallback. Verify evaluator defaults and actual requests as described - in SKILL.md Step 3; omission and explicit `null` are not interchangeable. -3. **Card vs. table** → re-read the card and surface discrepancies; only verified - evaluation settings qualify. Quickstarts, supported limits, and general - "Recommended Sampling" rows do not qualify without an explicit evaluation tie. -4. **Scope overrides to the stated tasks/modes**, including output caps. Do not - promote a task-specific or headline maximum to a suite-wide value. Apply - verified settings consistently to baseline and candidate when reproducing - their published comparison; otherwise confirm effective config parity. - -`provenance` — **`eval`** (20 rows): recorded as tied to evaluation; verify each -field against the card. **`rec`** (5 rows): inference recommendations, not -ordinary evaluation evidence. `max_num_tokens` records generation length, corresponding -to `nemo_evaluator_config.config.params.max_new_tokens`, not context length. +**The lookup and family priors below govern `temperature` / `top_p`, not output +budgets.** For `max_new_tokens`, follow SKILL.md Step 3's token-budget rule; +verify table values against the card and the applicable evaluation. + +**The card is the source of truth; this table is a reference, not a constraint.** +Use it to confirm what you read, to fill a gap when the card is silent, and as a +sanity check when you are unsure — never to override a value the card states. + +1. **Exact row, resolved per field.** `eval` → use it, cite the row. `rec` → use + it, but note in the config comment that it is recommended sampling, not a + stated eval setting; if a same-family `eval` row disagrees, surface both. + `—` → that field is unpublished; resolve **it alone** via step 2. +2. **No row** (new or unreleased variant, non-NVIDIA baseline, pre-2026) → take + the nearest same-family rows as the expected value. +3. **Card vs. table.** Agree → proceed. Card silent + family consistent → adopt + the sampling value and cite this file in a line comment. + **Card disagrees → the card wins**, but + surface it — defaults shift between generations, so a mismatch means re-read, + not auto-correct. +4. **Baseline and candidate share one setting.** Cards report both precisions + measured under the single setting listed; use the NVFP4 row for both. + +**Per-task sampling is precedent, not mandate.** Some notes record a +benchmark-specific `temperature` / `top_p` (Qwen3.6 SciCode `0.6`; Qwen3.6-27B +τ²-Bench Telecom `0.0` / `top_p=1.0`; Kimi-K3 `top_p=1.0` agentic). Engineers do +tune sampling per benchmark, so **follow the card you are working from** and use +these as the cross-check. Where the two disagree, **escalate to the user on a +regime change, not a nudge** — greedy (`temperature ≤ 0.1` or `top_p ≤ 1e-4`) +versus sampled flips the regime and materially moves both score and variance; +`0.95` vs `1.0` does not. NEL accepts per-task `temperature` / `top_p` under +`evaluation.tasks.*.nemo_evaluator_config`; only `max_new_tokens` is barred +(SKILL.md Step 3). + +> **Never take sampling from a card's quickstart snippet.** +> `SamplingParams(temperature=0.8, top_p=0.95)` and `max_tokens=32` are +> boilerplate, repeated verbatim across unrelated models. Only *Benchmarked +> with…* / *evaluated with…* / *We evaluate the model using…* sentences, +> "Recommended Sampling" rows, and footnotes under the accuracy table count. + +`provenance` — **`eval`** (20 rows): card ties the values to its accuracy table, +authoritative. **`rec`** (5 rows): card recommends them for inference without +that tie. `max_num_tokens` is the max generation length, i.e. +`nemo_evaluator_config.config.params.max_new_tokens`. | Model card ID | temp | top_p | max_num_tokens | prov | notes | | --- | --- | --- | --- | --- | --- | @@ -70,6 +87,26 @@ to `nemo_evaluator_config.config.params.max_new_tokens`, not context length. | `nvidia/NVIDIA-Nemotron-Labs-3-Elastic-30B-A3B-NVFP4` | 1.0 | 1.0 | — | rec | reasoning tasks | | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` | 0.6 think / 0.2 instruct | 0.95 think / — | 20480 think / 1024 instruct | rec | think adds `reasoning_budget=16384`, `grace_period=1024`; instruct sets `top_k=1` | +## Priors (verify against the card) + +- **`1.0 / 0.95` is the house default** — GLM 4.7–5.2, Kimi K2.6–K3, MiniMax + M2.5–M3, Gemma 4, Nemotron 3/3.5, Qwen3.6. Best guess when a recent card is + silent. +- **DeepSeek is carved out of it** — its row uses `top_p=1.0`, not `0.95`. + Never carry the house default onto an unlisted DeepSeek variant. +- **Qwen splits by variant** — thinking `0.6 / 0.95` at Qwen3.5, raised to `1.0` + at Qwen3.6; instruct/coder near-greedy `0 / 1e-5` with `16384`. +- **GLM** — `1.0 / 0.95` throughout; cap fell from 131072 (4.7, 5) to 64000 + (5.1, 5.2). +- **Caps cluster at 64000 / 65536 / 81920 / 128000 / 131072**, 64000 most + common. `16384` appears only with greedy instruct Qwen; DeepSeek-V4-Flash's + `384000` is a long-context outlier. +- **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench + Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench + (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`; conflicting + benchmark-specific budgets require separate configs or user confirmation, + not taking the maximum. + ## Refreshing Built from the HF API, verified to match the website pagination page for page diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 202a6c4c51d..3a33a6af053 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -10,8 +10,7 @@ "Runs 'nel skills build-config' with correct flags matching user answers, using NEL's current model_type terminology for the chat/reasoning template family", "Searches the web for the model card on HuggingFace and extracts model-specific settings", "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", - "Applies authoritative task policy first (including the scoped AA exception if reproducing AA); otherwise overrides card params only with applicable evaluation provenance or preserves verified checkpoint/server defaults", - "Honors explicit user/task generation requirements ahead of model-card settings and cites the source of each override", + "Sets reasoning sampling params from model card: temperature=1.0, top_p=1.0", "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)", "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false", "Adds deployment.pre_cmd using curl (not wget) to download nano_v3_reasoning_parser.py from HuggingFace", @@ -44,8 +43,7 @@ "Auto-detects quantization format by reading ./llama-3.1-8b-fp8/config.json first, then ./llama-3.1-8b-fp8/hf_quant_config.json if needed", "Finds quant_algo=FP8 and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", "Recommends accuracy-sensitive benchmarks from references/quantization-benchmarks.md", - "Reads the Llama-3.1-8B model card and checkpoint config.json/generation_config.json; distinguishes evaluation sampling from inference recommendations and context limits", - "Does not fill missing evaluation settings from same-family rows or generic 65536/16384 budgets; verifies effective evaluator requests rather than assuming omission or null preserves server defaults", + "Searches web for Llama-3.1-8B model card and extracts sampling params, context length, TP settings", "Fills in remaining missing values by asking user", "Runs dry-run, then a limited-samples canary with limit_samples=10, then full evaluation only after canary log checks pass", "Tunes parallelism from the canary and avoids over-raising it when benchmarks depend on inference judges or code execution containers", @@ -65,8 +63,7 @@ "Runs nel skills build-config with --execution slurm --deployment vllm --benchmarks standard", "Detects FP8 quantization from config.json or hf_quant_config.json and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", "Reads references/quantization-benchmarks.md and recommends accuracy-sensitive benchmarks", - "Researches model-card evaluation provenance and context length; only applies generation overrides backed by explicit user/task requirements or applicable benchmark statements", - "Keeps benchmark-specific settings scoped to their tasks instead of choosing the highest value anywhere in the card", + "Uses WebSearch to research model card for sampling params and context length", "Fills in SLURM-specific values: hostname, account, partition from user input", "Runs dry-run validation and a limited-samples canary before full evaluation", "Checks canary logs for judge, container, deployment, and code-execution failures before treating the run as ready", @@ -75,43 +72,6 @@ "Provides SSH-based log monitoring commands for SLURM execution" ] }, - { - "name": "aa-generation-policy", - "skills": ["evaluation"], - "query": "Prepare an AA reproduction config for a reasoning model on GPQA and AA-LCR v1.1, without submitting. The exact model lab recommends temperature 0.7 and discloses a maximum output of 98304 tokens, but neither statement mentions evaluation. It supports 256K context. There is no top_p guidance. Keep these benchmark versions.", - "files": [], - "expected_behavior": [ - "Reads the canonical AA methodology reference and treats applicable AA policy as authoritative ahead of generic evaluation-only card rules", - "Uses the lab-recommended temperature 0.7 and creator-disclosed reasoning output maximum 98304 without demanding an evaluation-provenance sentence; cites both disclosures", - "Explains that absent a lab temperature recommendation AA uses 0 for non-reasoning and 0.6 for reasoning", - "Does not invent top_p=0.95, a generic reasoning 65536 cap, or universal null; verifies effective requests and required budgets after adapters", - "Treats AA-LCR v1.1's 128K as a minimum, not an exact 131072 override; checks prompt plus output capacity and reports conflicts", - "Verifies retry semantics for up to 30 total API attempts including nested retries, rather than blindly setting max_retries=30", - "Scopes AA settings to applicable benchmark/version reproduction, preserves versions, and updates export provenance and tags" - ] - }, - { - "name": "aa-nonreasoning-output", - "skills": ["evaluation"], - "query": "Prepare non-reasoning AA GPQA reproduction configs for two models with no lab temperature recommendation. One supports 16384 output tokens; the other has a disclosed 8192 output cap. Both have ample context. Do not submit.", - "files": [], - "expected_behavior": [ - "Uses AA temperature 0 and output 16384 for the first model, reducing output to 8192 for the second with a cited capacity constraint", - "Does not require an evaluation-provenance sentence to apply AA methodology or spread the reduced cap to unrelated models/tasks", - "Does not impose a general AA top_p or exact 131072 context override; verifies effective requests" - ] - }, - { - "name": "non-aa-gpqa-defaults", - "skills": ["evaluation"], - "query": "Prepare a generic GPQA evaluation, not an AA reproduction. My reasoning model card has only general inference recommendations and supported output limits, with no evaluation settings. Do not submit.", - "files": [], - "expected_behavior": [ - "Does not infer AA reproduction from GPQA or its prompt format and does not apply the AA lab-recommendation exception", - "Preserves verified checkpoint/server defaults rather than imposing temperatures 0.6 or 1, top_p 0.95, output 16384 or 65536, a creator maximum, or null", - "Inspects evaluator defaults and effective requests; cites applicable evaluation evidence before any card-derived override" - ] - }, { "name": "scicode-eight-run-average", "skills": ["evaluation"], diff --git a/plugins/modelopt/skills/launching-evals/references/analyze-results.md b/plugins/modelopt/skills/launching-evals/references/analyze-results.md index d77aa78a799..ee51edb0513 100644 --- a/plugins/modelopt/skills/launching-evals/references/analyze-results.md +++ b/plugins/modelopt/skills/launching-evals/references/analyze-results.md @@ -43,11 +43,11 @@ Check logs for silent errors that may invalidate results: ## Step 3: Validate config and methodology 1. **Methodology consistency**: Verify same benchmark versions, prompt templates, sampling params, and infrastructure across all models. Flag discrepancies. -2. **Generation settings**: For applicable AA benchmark/version reproduction, validate against the [AA task policy](../../evaluation/references/aa-methodology.md), including its provenance exception. Otherwise follow the evaluation skill's normal user/task → evaluation-provenanced card → checkpoint/vLLM defaults policy. -3. **Reasoning model validation**: For AA reproduction, verify the creator-disclosed maximum output and AA temperature policy, not generic 64K or `null`. Otherwise verify effective settings without imposing generic sampling or budgets. Verify the intended reasoning mode. +2. **HF model card compliance**: Read the model's HuggingFace model card. Flag any deviations in inference parameters (temperature, top_p, max_new_tokens, deployment args, reasoning flags, etc.). +3. **Reasoning model validation**: Verify temp > 0, top_p > 0; `max_tokens` / `max_new_tokens` uses the creator-disclosed maximum output length, unless the model card specifies a budget used for the applicable evaluation (see `../../evaluation/SKILL.md`, Step 3). NOTE: `use_reasoning: False` in adapter_config does NOT mean reasoning is disabled — it only controls the reasoning interceptor. Whether reasoning is active depends on the model's own controls (deployment args, system prompt, API payload fields, etc.). -4. **Non-reasoning and effective requests**: For AA reproduction, verify `max_tokens` = `16384`, reduced only for smaller context/output limits per the AA policy. Otherwise preserve the normal default policy. Check evaluator/adapter defaults and actual requests; omission and `null` are not interchangeable. -5. **Max model length**: Check prompt + output capacity, including multi-turn history, against the supported context window and applicable task minimum (AA-LCR v1.1: 128K). Do not infer generation limits from `max_position_embeddings` or require exactly 131072 for every run. +4. **Non-reasoning model validation**: Verify `max_tokens` / `max_new_tokens` = 16384, lowered for smaller context/output caps, unless the model card specifies a budget used for the applicable evaluation. +5. **Max model length**: Verify `max-model-len` = 131072 (leaderboard-recommended). Long context benchmarks (AA LCR, RULER) and agentic benchmarks may require a longer `max-model-len`. 6. **RULER tasks**: Check thinking disabled, walltime=4h, rope-scaling for Qwen models 7. **AA baseline comparison**: Compare results against Artificial Analysis published scores. Exact match not expected — flag significant deviations. 8. **Model baseline comparison**: For quantized runs, compare results against the matching baseline model run when available. The baseline may be unquantized or simply less quantized (for example, FP8 as the baseline for NVFP4). Use the same benchmark version, task config, serving args, token limits, dataset setup, and infrastructure before treating the delta as a quantization effect. From 9fe55aa4645aca98487ddea5e17d79903de2039e Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Thu, 17 Sep 2026 18:58:36 +0000 Subject: [PATCH 5/6] docs: remove unknown output-budget clarification prompt Signed-off-by: Chad Voegele --- plugins/modelopt/skills/evaluation/SKILL.md | 2 +- .../skills/evaluation/references/model-card-research.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 9fb70bc2857..ad4b7b03c99 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -329,7 +329,7 @@ overridden task is reported under sampling params it did not use. #### `max_new_tokens` — mandatory model-card lookup 1. **Read the HF model card before setting the value.** Identify the model's reasoning mode, creator-disclosed maximum output length, and any `max_tokens` / `max_new_tokens` used for the applicable evaluation. Cite the source and rationale in a config comment; quickstart examples are not evaluation budgets. -2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum; ask the user if it remains unknown. +2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum. 3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs or user confirmation. 4. Verify that the output budget fits alongside the input, including accumulated multi-turn history. Surface conflicts rather than silently clipping a creator/evaluation budget. Check `finish_reason: length` after the run (`references/run-validation.md`); a higher cap does not fix runaway reasoning. diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index 5d559cfe4f5..42a0a6eb23c 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -20,7 +20,7 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca creator-disclosed maximum output length and any budget used for the applicable evaluation, with sources. Non-reasoning defaults to **16384**, lowered for smaller context/output caps; reasoning uses the creator-disclosed output - maximum (ask if unknown). An explicit model-card budget for the applicable + maximum. An explicit model-card budget for the applicable evaluation can override these defaults. No highest-number or same-family fallback. Keep one top-level value; conflicting benchmark budgets require separate configs or user confirmation. See SKILL.md Step 3 From 637a1026cd4078e5879cd14871132a5442251288 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Thu, 17 Sep 2026 18:59:52 +0000 Subject: [PATCH 6/6] docs: remove output-budget user confirmation instructions Signed-off-by: Chad Voegele --- plugins/modelopt/skills/evaluation/SKILL.md | 2 +- .../skills/evaluation/references/model-card-research.md | 2 +- .../skills/evaluation/references/nvfp4-modelcard-sampling.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index ad4b7b03c99..6c848b45316 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -330,7 +330,7 @@ overridden task is reported under sampling params it did not use. 1. **Read the HF model card before setting the value.** Identify the model's reasoning mode, creator-disclosed maximum output length, and any `max_tokens` / `max_new_tokens` used for the applicable evaluation. Cite the source and rationale in a config comment; quickstart examples are not evaluation budgets. 2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum. -3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs or user confirmation. +3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs. 4. Verify that the output budget fits alongside the input, including accumulated multi-turn history. Surface conflicts rather than silently clipping a creator/evaluation budget. Check `finish_reason: length` after the run (`references/run-validation.md`); a higher cap does not fix runaway reasoning. #### Quantization-aware benchmark defaults diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index 42a0a6eb23c..d76778155e4 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -23,7 +23,7 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca maximum. An explicit model-card budget for the applicable evaluation can override these defaults. No highest-number or same-family fallback. Keep one top-level value; conflicting benchmark budgets require - separate configs or user confirmation. See SKILL.md Step 3 + separate configs. See SKILL.md Step 3 "`max_new_tokens` — mandatory model-card lookup" for the full rule. - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed) - Reasoning config (if applicable): diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md index a47d9f12a0f..0230951df4a 100644 --- a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -104,7 +104,7 @@ that tie. `max_num_tokens` is the max generation length, i.e. - **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`; conflicting - benchmark-specific budgets require separate configs or user confirmation, + benchmark-specific budgets require separate configs, not taking the maximum. ## Refreshing