From 664614d37b7d4f765d7564c010547adeddabcdba Mon Sep 17 00:00:00 2001 From: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Date: Tue, 15 Sep 2026 21:36:58 +0000 Subject: [PATCH 1/3] docs: add single-GPU PTQ announcement blog Adds docs/source/announcements/single-gpu-ptq.rst covering the layerwise calibration + per-layer shard export workflow, with Kimi-K3 on a single B200 as the headline example, plus an index card. Frames the problem as the memory floor for calibration rather than model size: layerwise calibration and per-layer export drop that floor from one model to one layer. Results carry the merged DeepSeek-R1 and Nemotron-Ultra numbers; the Kimi-K3 row is marked TODO(reconfirm) pending a full run on the current exporter. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> --- docs/source/announcements/single-gpu-ptq.rst | 205 +++++++++++++++++++ docs/source/index.rst | 7 + 2 files changed, 212 insertions(+) create mode 100644 docs/source/announcements/single-gpu-ptq.rst diff --git a/docs/source/announcements/single-gpu-ptq.rst b/docs/source/announcements/single-gpu-ptq.rst new file mode 100644 index 00000000000..73c84f6ef87 --- /dev/null +++ b/docs/source/announcements/single-gpu-ptq.rst @@ -0,0 +1,205 @@ +:orphan: + +Quantizing a 1.5 TB Model on a Single GPU +######################################### + +:Author: Model Optimizer Team +:Date: September 15, 2026 +:Tags: quantization, nvfp4, layerwise, moe, single-gpu, modelopt + +Post-training quantization needs calibration, and calibration needs forward passes over real +data. Traditionally that has meant holding the whole model in accelerator memory — so the +hardware floor for *quantizing* a model has been roughly the floor for *serving* it. For a +large mixture-of-experts checkpoint, that puts an otherwise routine PTQ run behind a +multi-node allocation. + +`Model Optimizer `_ now calibrates and exports one +decoder layer at a time. The memory a calibration run needs is set by the largest layer, not +by the model, and the floor drops from *one model* to *one layer*. + +As an existence proof: ``moonshotai/Kimi-K3`` — a 1.5 TB checkpoint with 896 experts across +93 layers — quantized to NVFP4 on a **single B200**. + +The memory floor for calibration +******************************** + +The constraint worth naming is not that models are large. It is that calibration has been +*all-or-nothing*: the forward pass wants every layer resident, even though it only ever reads +one layer at a time. Two consequences follow, and both cost hardware rather than accuracy. + +First, the whole checkpoint has to fit somewhere fast. Second, the classic PTQ shape — +calibrate the whole model, then export the whole model — traverses it twice, so a completed +calibration still owes a second full pass before there is a checkpoint on disk. + +Neither is inherent to the math. Calibration statistics for layer *i* depend on the +activations entering layer *i*, which the previous layer already produced. If those +activations are carried forward explicitly, layers can be visited strictly one at a time, and +the resident set never has to exceed one of them. + +One layer at a time +******************* + +Three pieces make that concrete. + +**Weights spill to disk.** An ``accelerate`` device map with explicit GPU and CPU budgets +keeps the bulk of the checkpoint on disk or in host RAM and materializes only what the +current step touches. + +**Calibration walks layer by layer.** ``layerwise`` calibration runs the forward pass one +decoder layer at a time, caching the activations at each boundary so the next layer has its +input without replaying the ones before it. + +**Each layer is exported the moment it is finished.** With ``layerwise.export_dir`` set, a +layer is quantized and written to its own checkpoint shard as soon as calibration is done +with it — and then released. + +That third piece is the one worth remembering, because it collapses two problems into one +artifact: + +.. note:: + + **The shards are the resume artifact.** There is no separate full-precision scratch copy + accumulating beside the run, and no second whole-model export pass owed at the end. A + shard on disk *is* the record that its layer is done. + +Running it +********** + +Setting the config field is the entire switch — there is no CLI flag: + +.. code-block:: yaml + + quantize: + algorithm: + method: max + layerwise: + enable: true + calib_mutates_weights: false # amax-only fast path + export_dir: /tmp/modelopt_layerwise_export # presence is the switch; + # value is replaced with --export_path + # checkpoint_dir omitted -> derived as .layerwise_resume + +Then the run itself: + +.. code-block:: bash + + python examples/hf_ptq/hf_ptq.py \ + --pyt_ckpt_path \ + --recipe models/moonshotai/Kimi-K3/ptq/nvfp4_experts-kv_none_layerwise_export \ + --export_path \ + --qformat nvfp4 --trust_remote_code --attn_implementation eager \ + --offload_folder --max_gpu_memory_gb 140 --max_cpu_memory_gb 1700 \ + --calib_size 256 --batch_size 8 --skip_generate + +What you need on hand: one GPU, a GPU and CPU memory budget you choose, and fast scratch +sized for the checkpoint you are writing. + +A few configurations are refused *before* calibration starts, rather than producing a quietly +different checkpoint — worth checking against your model before you spend a session: + +* AWQ and SVDQuant, which need whole-model pre-quant-scale steps +* Models with tied weights (``tie_word_embeddings``); use ``export_hf_checkpoint()`` instead +* Multi-process jobs such as FSDP2, where every rank would write the same shards + +Per-layer export also leaves the in-memory model in export form, so ``hf_ptq.py`` sets +``--skip_generate`` for you. + +Interrupt it +************ + +Because a committed shard means a finished layer, resume needs no special invocation: rerun +the identical command. Finished layers are skipped, and calibration picks up at the boundary +it last committed. + +.. code-block:: text + + Checkpoint: resuming layerwise calibration from layer 13/93 + +The Kimi-K3 checkpoint was produced this way across three separate four-hour GPU sessions. +Resume is exact rather than approximate: a run killed with ``SIGKILL`` after 25 of 48 layers +and then resumed produced a checkpoint identical, tensor for tensor, to the uninterrupted run. + +Results +******* + +.. warning:: + + **Draft:** the Kimi-K3 row is pending reconfirmation against a full run on the current + exporter. Figures marked ``TODO(reconfirm)`` must be replaced from that run before publish. + +.. list-table:: + :header-rows: 1 + + * - Model + - Layers + - GPU budget + - Wall clock + - Peak GPU + - Output + * - Kimi-K3 (1.5 TB) + - 93 + - 140 GB + - ``TODO(reconfirm)`` + - ``TODO(reconfirm)`` + - 93 layer shards + tail + index + * - DeepSeek-R1 671B (642 GB) + - 61 + - 80 GB + - 40 min 12 s + - 88.9 GB + - 403 GB, 40 shards + * - Nemotron-3-Ultra 550B (~1.1 TB) + - 108 + - 80 GB + - 47 min 16 s + - 76.7 GB + - 365 GB, 34 shards + +Every Kimi-K3 expert projection — ``TODO(reconfirm)``, or 92 × 896 × 3 — carries a calibrated +``input_scale``, and vLLM selects the FlashInfer TRT-LLM NVFP4 MoE kernel rather than the +emulation fallback. + +Resume state stays bounded, because only the committed boundary's activations are kept: 332 KB +beside 22 GB of shards on a 35B model, 396 KB beside 19 GB on a 30B. + +The correctness claim behind these runs is narrow and worth stating precisely: **per-layer +export produces the same checkpoint as whole-model export.** Across four models, the two paths +were compared tensor for tensor and config for config — 123,513 tensors on a 35B MoE, 74,163 +on a 30B, 0 mismatched — with every ``weight_map`` entry resolving to the shard that actually +holds it. Under vLLM, checkpoints exported both ways produce identical greedy generations. + +What you're trading +******************* + +Each of these is a consequence of the design rather than a defect, so the useful question is +whether the trade fits your constraints. + +**Time.** The layer walk is sequential by construction: roughly 40–47 minutes for models in +the 550–671B range on one GPU. You are trading wall clock for hardware, which is the point, +but it is a real cost on a large model. + +**Calibration algorithms are restricted — for now.** ``calib_mutates_weights: false``, the +flag that makes resume cheap, is currently whitelisted to amax-only methods: max, MSE, and +local Hessian. Weight-mutating calibration (GPTQ, AWQ, SmoothQuant) and AutoQuantize are +refused today. That restriction is conservative rather than fundamental — under per-layer +export the mutated weights land in the layer's shard before the layer is released — so GPTQ in +particular is expected to need little or no change. Gradual enablement is on the way. Formats +today are FP8 and NVFP4. + +**It solves calibration memory, not serving memory.** The checkpoint this produces still has +to be served, and whether your hardware can serve it is a separate question this workflow does +not answer. + +**It scales in depth, not width.** Peak memory here is a lower bound, not a knob: the GPU +still has to hold one decoder layer, plus its activations, at once. A model with many modest +layers is easy; a model with one enormous layer is the boundary this design cannot move. That +is also how to predict whether your model fits before spending a session finding out — divide, +don't guess. + +Resources +********* + +* `Single-GPU disk-offload PTQ (PR #2008) `_ +* `Per-layer shard export (PR #2136) `_ +* `Multimodal and MTP support for layerwise export (PR #2303) `_ +* `Kimi-K3 on layerwise fused export (PR #2218) `_ diff --git a/docs/source/index.rst b/docs/source/index.rst index 0f2923a52d6..d89ce3c4ea7 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -13,6 +13,7 @@ Release notes, technical updates, examples, and deployment stories from the Mode + @@ -24,6 +25,12 @@ Release notes, technical updates, examples, and deployment stories from the Mode
+
+
September 15, 2026 · Model Optimizer Team
+

Quantizing a 1.5 TB Model on a Single GPU

+

Layerwise calibration and per-layer shard export drop the memory floor for PTQ from one model to one layer.

+
quantizationnvfp4layerwisemoesingle-gpumodelopt
+
August 24, 2026 · Model Optimizer Team

AutoQuantize: A Fast Automatic Mixed-Precision Assignment

From ff6408095a1712a0de2ee985bb42483fd5190af0 Mon Sep 17 00:00:00 2001 From: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Date: Tue, 15 Sep 2026 22:26:36 +0000 Subject: [PATCH 2/3] docs: tighten single-GPU PTQ blog after content pass - Halve "The memory floor for calibration"; drop the enumeration and the closing restatement. - Explain the mechanism as a loop interchange, with before/after pseudocode, why the swap makes a finished layer well defined, and the boundary activations it costs. - Results: Kimi-K3 only. Drop the DeepSeek-R1 and Nemotron-Ultra rows, whose calibration settings are not comparable. Add why peak GPU exceeds the GPU budget and why host RSS spikes above the CPU budget during load: both flags size the accelerate device map, not total allocation. - Drop the calib_mutates_weights framing from the trade-offs; note that a PR extending the workflow to GPTQ and friends is on the way. - Remove the Resources section. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> --- docs/source/announcements/single-gpu-ptq.rst | 120 ++++++++++--------- 1 file changed, 66 insertions(+), 54 deletions(-) diff --git a/docs/source/announcements/single-gpu-ptq.rst b/docs/source/announcements/single-gpu-ptq.rst index 73c84f6ef87..e073cf2b85a 100644 --- a/docs/source/announcements/single-gpu-ptq.rst +++ b/docs/source/announcements/single-gpu-ptq.rst @@ -25,36 +25,55 @@ The memory floor for calibration The constraint worth naming is not that models are large. It is that calibration has been *all-or-nothing*: the forward pass wants every layer resident, even though it only ever reads -one layer at a time. Two consequences follow, and both cost hardware rather than accuracy. +one layer at a time. The classic PTQ shape compounds it — calibrate the whole model, then +export the whole model — so a finished calibration still owes a second full traversal before +any checkpoint exists on disk. -First, the whole checkpoint has to fit somewhere fast. Second, the classic PTQ shape — -calibrate the whole model, then export the whole model — traverses it twice, so a completed -calibration still owes a second full pass before there is a checkpoint on disk. - -Neither is inherent to the math. Calibration statistics for layer *i* depend on the -activations entering layer *i*, which the previous layer already produced. If those -activations are carried forward explicitly, layers can be visited strictly one at a time, and -the resident set never has to exceed one of them. +Neither is inherent to the math. Layer *i*'s calibration statistics depend only on the +activations entering layer *i*, which layer *i-1* already produced. One layer at a time ******************* -Three pieces make that concrete. +The mechanism is a loop interchange. Conventional calibration puts data on the outside and +depth on the inside, so every layer has to be resident for every batch: + +.. code-block:: python + + for batch in calib_data: # outer: data + h = embed(batch) + for layer in model.layers: # inner: depth + h = layer(h) # all 93 layers live, the whole time + +Layerwise calibration swaps the two loops — depth outside, data inside: + +.. code-block:: python + + acts = [embed(batch) for batch in calib_data] # activations at the boundary + for layer in model.layers: # outer: depth + for i, h in enumerate(acts): # inner: data + acts[i] = layer(h) # one layer live at a time + calibrate(layer); quantize(layer); export(layer); release(layer) + +The interchange is what buys everything else. After the swap a layer is *finished* the moment +its inner loop ends — every batch it will ever see has already been through it — so the four +calls on that last line are well defined, and a shard written there is a truthful record that +its layer is done. + +The price is the boundary. Instead of one activation tensor in flight per batch, the whole +calibration set's activations are held between layers. That is a real cost, but it is bounded +by the calibration set rather than by the model. + +Two pieces turn that into a run. **Weights spill to disk.** An ``accelerate`` device map with explicit GPU and CPU budgets keeps the bulk of the checkpoint on disk or in host RAM and materializes only what the current step touches. -**Calibration walks layer by layer.** ``layerwise`` calibration runs the forward pass one -decoder layer at a time, caching the activations at each boundary so the next layer has its -input without replaying the ones before it. - **Each layer is exported the moment it is finished.** With ``layerwise.export_dir`` set, a layer is quantized and written to its own checkpoint shard as soon as calibration is done -with it — and then released. - -That third piece is the one worth remembering, because it collapses two problems into one -artifact: +with it — and then released. That is the piece worth remembering, because it collapses two +problems into one artifact: .. note:: @@ -124,36 +143,38 @@ Results .. warning:: - **Draft:** the Kimi-K3 row is pending reconfirmation against a full run on the current - exporter. Figures marked ``TODO(reconfirm)`` must be replaced from that run before publish. + **Draft:** these figures are pending reconfirmation against a full run on the current + exporter. Everything marked ``TODO(reconfirm)`` must be replaced from that run before + publish. .. list-table:: :header-rows: 1 - * - Model - - Layers - - GPU budget - - Wall clock - - Peak GPU - - Output - * - Kimi-K3 (1.5 TB) + * - Kimi-K3 + - + * - Layers - 93 + * - GPU budget (``--max_gpu_memory_gb``) - 140 GB + * - CPU budget (``--max_cpu_memory_gb``) + - 1700 GB + * - Wall clock + - ``TODO(reconfirm)`` + * - Peak GPU - ``TODO(reconfirm)`` + * - Peak RSS - ``TODO(reconfirm)`` + * - Output - 93 layer shards + tail + index - * - DeepSeek-R1 671B (642 GB) - - 61 - - 80 GB - - 40 min 12 s - - 88.9 GB - - 403 GB, 40 shards - * - Nemotron-3-Ultra 550B (~1.1 TB) - - 108 - - 80 GB - - 47 min 16 s - - 76.7 GB - - 365 GB, 34 shards + +**Those budgets are weight-placement budgets, not caps**, and it is worth knowing that before +you size them. ``--max_gpu_memory_gb`` and ``--max_cpu_memory_gb`` feed ``accelerate``'s device +map: they decide how much of the *checkpoint* is assigned to each device, and everything the +run allocates on top of the weights falls outside them. So expect peak GPU to land somewhat +above the GPU budget — activations, calibration buffers and the CUDA context are not counted +against it — and expect a much larger transient spike in host RSS while shards are read and +dispatched, settling to a far lower steady state once the offload folder is populated. Size +both with headroom rather than to the exact capacity of the machine. Every Kimi-K3 expert projection — ``TODO(reconfirm)``, or 92 × 896 × 3 — carries a calibrated ``input_scale``, and vLLM selects the FlashInfer TRT-LLM NVFP4 MoE kernel rather than the @@ -178,13 +199,12 @@ whether the trade fits your constraints. the 550–671B range on one GPU. You are trading wall clock for hardware, which is the point, but it is a real cost on a large model. -**Calibration algorithms are restricted — for now.** ``calib_mutates_weights: false``, the -flag that makes resume cheap, is currently whitelisted to amax-only methods: max, MSE, and -local Hessian. Weight-mutating calibration (GPTQ, AWQ, SmoothQuant) and AutoQuantize are -refused today. That restriction is conservative rather than fundamental — under per-layer -export the mutated weights land in the layer's shard before the layer is released — so GPTQ in -particular is expected to need little or no change. Gradual enablement is on the way. Formats -today are FP8 and NVFP4. +**Calibration algorithms are restricted — for now.** The workflow supports max, MSE and local +Hessian calibration today, in FP8 and NVFP4. Weight-mutating calibration (GPTQ, AWQ, +SmoothQuant) and AutoQuantize are refused. That restriction is conservative rather than +fundamental: under per-layer export a mutated weight is already written into the layer's shard +before the layer is released, so the machinery is in the right shape for it. A PR extending the +workflow to GPTQ and friends is on the way. **It solves calibration memory, not serving memory.** The checkpoint this produces still has to be served, and whether your hardware can serve it is a separate question this workflow does @@ -195,11 +215,3 @@ still has to hold one decoder layer, plus its activations, at once. A model with layers is easy; a model with one enormous layer is the boundary this design cannot move. That is also how to predict whether your model fits before spending a session finding out — divide, don't guess. - -Resources -********* - -* `Single-GPU disk-offload PTQ (PR #2008) `_ -* `Per-layer shard export (PR #2136) `_ -* `Multimodal and MTP support for layerwise export (PR #2303) `_ -* `Kimi-K3 on layerwise fused export (PR #2218) `_ From 57295868b00b8324bd7979fc64f9c89d3b08aeae Mon Sep 17 00:00:00 2001 From: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Date: Tue, 15 Sep 2026 23:01:44 +0000 Subject: [PATCH 3/3] docs: name Kimi-K3 in the title and state the MXFP4 source caveat Title becomes "Quantizing a 1.5 TB Kimi-K3 Model on a Single GPU", updated in the page heading and both index-card fields. Adds a note after the run command recording that the published Kimi-K3 checkpoint stores its routed experts in MXFP4, and that this run dequantizes them to BF16 before calibrating, so the end-to-end path is MXFP4 -> BF16 -> NVFP4. The round trip cannot recover what the source quantization discarded; the example demonstrates the single-GPU workflow and does not evaluate that accuracy cost. This also explains the placeholder in the command. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> --- docs/source/announcements/single-gpu-ptq.rst | 13 +++++++++++-- docs/source/index.rst | 4 ++-- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/source/announcements/single-gpu-ptq.rst b/docs/source/announcements/single-gpu-ptq.rst index e073cf2b85a..2a4e7a054be 100644 --- a/docs/source/announcements/single-gpu-ptq.rst +++ b/docs/source/announcements/single-gpu-ptq.rst @@ -1,7 +1,7 @@ :orphan: -Quantizing a 1.5 TB Model on a Single GPU -######################################### +Quantizing a 1.5 TB Kimi-K3 Model on a Single GPU +################################################# :Author: Model Optimizer Team :Date: September 15, 2026 @@ -113,6 +113,15 @@ Then the run itself: What you need on hand: one GPU, a GPU and CPU memory budget you choose, and fast scratch sized for the checkpoint you are writing. +.. note:: + + **Kimi-K3 is used here to demonstrate the workflow, not as a recommended recipe.** The + published checkpoint stores its routed experts in MXFP4. This run dequantizes those weights + to BF16 and calibrates the BF16 model — hence ```` above — so the end-to-end path + is MXFP4 → BF16 → NVFP4. Dequantizing cannot recover what the original MXFP4 quantization + already discarded, and the accuracy cost of that round trip is out of scope here. What is + being shown is that a checkpoint this size can be calibrated and exported on one GPU. + A few configurations are refused *before* calibration starts, rather than producing a quietly different checkpoint — worth checking against your model before you spend a session: diff --git a/docs/source/index.rst b/docs/source/index.rst index d89ce3c4ea7..b88df73872c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -25,9 +25,9 @@ Release notes, technical updates, examples, and deployment stories from the Mode
-