diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index da57288528f..6c848b45316 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -309,13 +309,15 @@ nemo_evaluator_config: parallelism: ??? # Required — size per references/parallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear request_timeout: 3600 max_retries: 10 - max_new_tokens: 65536 # see rule below + max_new_tokens: ??? # resolve from the token-budget rule below temperature: 1.0 # from model card (reasoning); adjust top_p: 0.95 # from model card (reasoning); adjust ``` Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere. +**For `max_new_tokens`, the token-budget rule below takes precedence over table recommendations and family fallbacks.** + **Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` / `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's "Lookup" section. **`temperature` / `top_p` are different: per-task overrides ARE allowed and often required.** Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives `top_p = 0.95` for agentic scenarios and `1.0` otherwise, so a single top-level `0.95` is wrong for every non-agentic task. @@ -326,13 +328,10 @@ overridden task is reported under sampling params it did not use. #### `max_new_tokens` — mandatory model-card lookup -1. **Fetch the HF model card before writing the value.** Not optional. -2. Scan for any `max_tokens` / `max_new_tokens` / "output length" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment. - **Card figures are SINGLE-TURN.** On multi-turn / agentic benchmarks the model's own answer is fed back in, so the cap must satisfy `n_turns × max_new_tokens + prompt < max_model_len`. Taking a card's headline "384K output" literally lost SciCode samples to HTTP 400; 65536 was clean. (`references/run-validation.md` already covers checking `finish_reason: length` after a run.) -3. **Consult `references/nvfp4-modelcard-sampling.md` as a reference.** Listed and in agreement → proceed with confidence. Listed and different → **the card wins**; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its `max_num_tokens` column records the card's *headline* cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row. -4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user. -5. **Forbidden:** writing `max_new_tokens: ` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence. -6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence. +1. **Read the HF model card before setting the value.** Identify the model's reasoning mode, creator-disclosed maximum output length, and any `max_tokens` / `max_new_tokens` used for the applicable evaluation. Cite the source and rationale in a config comment; quickstart examples are not evaluation budgets. +2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum. +3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs. +4. Verify that the output budget fits alongside the input, including accumulated multi-turn history. Surface conflicts rather than silently clipping a creator/evaluation budget. Check `finish_reason: length` after the run (`references/run-validation.md`); a higher cap does not fix runaway reasoning. #### Quantization-aware benchmark defaults diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml index 1f8faf6def3..41b7c4c8c40 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml @@ -132,7 +132,7 @@ evaluation: parallelism: ??? # Number of concurrent requests per each benchmark request_timeout: 3600 max_retries: 10 - max_new_tokens: 65536 # 64K for reasoning models; use 16384 (16K) for non-reasoning; prefer model card value + max_new_tokens: ??? # Resolve per SKILL.md's token-budget rule; match the export description/tag below temperature: 1.0 # from model card (reasoning mode); adjust per card top_p: 0.95 # from model card (reasoning mode); adjust per card target: @@ -197,7 +197,7 @@ export: # would resolve. Cross-server, that makes a same-named experiment here and only # modelopt_run_url reaches the PTQ run (Step 4). experiment_name: ${oc.env:USER}/CHANGEME-served-model-name - description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=65536' + description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=???' log_logs: true log_artifacts: true only_required: false @@ -207,7 +207,7 @@ export: model: CHANGEME-served-model-name temperature: '1.0' top_p: '0.95' - max_new_tokens: '65536' + max_new_tokens: ??? # From the checkpoint's .experiment.json when present; drop when absent. Keep the # quotes (a bare 20260910 becomes a date) and skip any value containing ${ -- quoting # does not stop OmegaConf resolving it. diff --git a/plugins/modelopt/skills/evaluation/references/model-card-research.md b/plugins/modelopt/skills/evaluation/references/model-card-research.md index 80db5cce28b..d76778155e4 100644 --- a/plugins/modelopt/skills/evaluation/references/model-card-research.md +++ b/plugins/modelopt/skills/evaluation/references/model-card-research.md @@ -12,21 +12,19 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them, grouped by family. Required for any NVFP4 checkpoint or same-family sibling; it is also the - best source of a default when the card is silent. Pre-2026 releases are out + best source of a sampling default when the card is silent; output budgets + follow the rule below instead. Pre-2026 releases are out of scope there — read their cards. - Context length (`deployment.extra_args: "--max-model-len "`) -- **Output length (`max_new_tokens`) — mandatory extraction.** Scan the - card for any `max_tokens` / `max_new_tokens` / "output length" - recommendation. Cards often list two values (e.g., Qwen3.x: `32768` - thinking-general + `81920` math/coding). **Pick the highest value** and - apply at the top level (no per-task overrides). If the card is genuinely - silent on output length, note that explicitly and fall back to the - generic default (64K reasoning / 16K non-reasoning) — never write a - config with "card not yet checked" + generic default. Check - `nvfp4-modelcard-sampling.md` for the model or its family before falling - back; a same-family published cap beats the generic default. See SKILL.md - Step 3 "`max_new_tokens` — pick a single top-level value" for the full - rule. +- **Output length (`max_new_tokens`) — mandatory extraction.** Record the + creator-disclosed maximum output length and any budget used for the applicable + evaluation, with sources. Non-reasoning defaults to **16384**, lowered for + smaller context/output caps; reasoning uses the creator-disclosed output + maximum. An explicit model-card budget for the applicable + evaluation can override these defaults. No highest-number or same-family + fallback. Keep one top-level value; conflicting benchmark budgets require + separate configs. See SKILL.md Step 3 + "`max_new_tokens` — mandatory model-card lookup" for the full rule. - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed) - Reasoning config (if applicable): - reasoning on/off: use either: diff --git a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md index f1f91d80a60..0230951df4a 100644 --- a/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md +++ b/plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md @@ -15,6 +15,10 @@ is silent or ambiguous. It does not replace reading the card — see ## Lookup +**The lookup and family priors below govern `temperature` / `top_p`, not output +budgets.** For `max_new_tokens`, follow SKILL.md Step 3's token-budget rule; +verify table values against the card and the applicable evaluation. + **The card is the source of truth; this table is a reference, not a constraint.** Use it to confirm what you read, to fill a gap when the card is silent, and as a sanity check when you are unsure — never to override a value the card states. @@ -23,16 +27,11 @@ sanity check when you are unsure — never to override a value the card states. it, but note in the config comment that it is recommended sampling, not a stated eval setting; if a same-family `eval` row disagrees, surface both. `—` → that field is unpublished; resolve **it alone** via step 2. - `max_num_tokens` is the card's *headline* cap — where a note names a higher - per-task cap (GLM-5.2 GPQA `100000`, Qwen3.5-397B-V2 τ²-Telecom `128000`, - Kimi-K3 uncapped for Terminal-Bench) and that task is in your suite, SKILL.md - Step 3's take-the-highest rule governs the single top-level value, not the - column. 2. **No row** (new or unreleased variant, non-NVIDIA baseline, pre-2026) → take the nearest same-family rows as the expected value. 3. **Card vs. table.** Agree → proceed. Card silent + family consistent → adopt - the family value and cite this file in a line comment; that beats SKILL.md - Step 3's generic 65536 / 16384. **Card disagrees → the card wins**, but + the sampling value and cite this file in a line comment. + **Card disagrees → the card wins**, but surface it — defaults shift between generations, so a mismatch means re-read, not auto-correct. 4. **Baseline and candidate share one setting.** Cards report both precisions @@ -104,9 +103,9 @@ that tie. `max_num_tokens` is the max generation length, i.e. `384000` is a long-context outlier. - **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench - (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`, so when a card - lists two caps **take the maximum** as the single top-level value and note the - split in a comment. + (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`; conflicting + benchmark-specific budgets require separate configs, + not taking the maximum. ## Refreshing diff --git a/plugins/modelopt/skills/launching-evals/references/analyze-results.md b/plugins/modelopt/skills/launching-evals/references/analyze-results.md index 48e0a87bb59..ee51edb0513 100644 --- a/plugins/modelopt/skills/launching-evals/references/analyze-results.md +++ b/plugins/modelopt/skills/launching-evals/references/analyze-results.md @@ -44,9 +44,9 @@ Check logs for silent errors that may invalidate results: 1. **Methodology consistency**: Verify same benchmark versions, prompt templates, sampling params, and infrastructure across all models. Flag discrepancies. 2. **HF model card compliance**: Read the model's HuggingFace model card. Flag any deviations in inference parameters (temperature, top_p, max_new_tokens, deployment args, reasoning flags, etc.). -3. **Reasoning model validation**: Verify temp > 0, top_p > 0, `max_tokens` = null (allow full output length). +3. **Reasoning model validation**: Verify temp > 0, top_p > 0; `max_tokens` / `max_new_tokens` uses the creator-disclosed maximum output length, unless the model card specifies a budget used for the applicable evaluation (see `../../evaluation/SKILL.md`, Step 3). NOTE: `use_reasoning: False` in adapter_config does NOT mean reasoning is disabled — it only controls the reasoning interceptor. Whether reasoning is active depends on the model's own controls (deployment args, system prompt, API payload fields, etc.). -4. **Non-reasoning model validation**: Verify `max_tokens` = 16k +4. **Non-reasoning model validation**: Verify `max_tokens` / `max_new_tokens` = 16384, lowered for smaller context/output caps, unless the model card specifies a budget used for the applicable evaluation. 5. **Max model length**: Verify `max-model-len` = 131072 (leaderboard-recommended). Long context benchmarks (AA LCR, RULER) and agentic benchmarks may require a longer `max-model-len`. 6. **RULER tasks**: Check thinking disabled, walltime=4h, rope-scaling for Qwen models 7. **AA baseline comparison**: Compare results against Artificial Analysis published scores. Exact match not expected — flag significant deviations.