QuantBridge

MXFP4 · NVFP4 · MXFP8 · FP8 ⇄ FP16 → Q/IQ GGUF — one guarded command

Detect the checkpoint, choose a maintained backend, preflight the route, and never overwrite a model.

Quick start · Examples · Compatibility · Publish

---

QuantBridge is a small orchestration layer for model conversion. It does not pretend that every file called “MXFP4” has the same byte layout. It reads the checkpoint metadata and delegates each layout to the project that owns it:

Job	Backend selected by auto
Dense local HF Safetensors → FP16	Built-in streaming converter
HF MXFP/FP8/compressed checkpoint → FP16	Transformers + the checkpoint's registered dequantizer
MLX MXFP4/NVFP4/MXFP8 → FP16	mlx-lm
FP16 → MXFP4/NVFP4/MXFP8/FP8	AutoRound, or MLX on Apple silicon
HF/MXFP/NVFP → Q4/Q6/Q8/IQ3/IQ4 GGUF	Current llama.cpp converter + llama-quantize
GGUF → another GGUF quant	llama-quantize with explicit requantization opt-in when needed

Unsupported layouts fail with the original backend error. QuantBridge never invents a scale layout or silently treats packed bytes as ordinary floats.

Quick start

1. Clone and create an environment

Linux/macOS:

git clone https://github.com/verkiki/quantbridge.git
cd quantbridge
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Windows PowerShell:

git clone https://github.com/verkiki/quantbridge.git
cd quantbridge
py -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e .

2. Install only the backend you need

For dequantizing serialized Hugging Face MXFP/NVFP/FP8 checkpoints:

python -m pip install -e ".[transformers]"

For creating MXFP4, NVFP4, MXFP8, or FP8 checkpoints on Linux/CUDA:

python -m pip install -e ".[quant]"

For MLX conversion on Apple silicon:

python -m pip install -e ".[mlx]"

For GGUF, use a current llama.cpp checkout:

git clone https://github.com/ggml-org/llama.cpp.git
python -m pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --config Release -j

Then either pass --llama-cpp /path/to/llama.cpp or set LLAMA_CPP.

3. Check the installation

quantbridge doctor --llama-cpp /path/to/llama.cpp
quantbridge inspect /models/source
quantbridge list-targets

Everyday commands

MXFP/NVFP/FP8 → FP16

quantbridge convert /models/source /models/source-fp16 --to fp16

For a Hub model:

quantbridge convert OWNER/MODEL ./MODEL-fp16 --to fp16

The result is a dense FP16 checkpoint reconstructed from the values in the quantized source. It is not the original pre-quantization checkpoint.

MXFP/NVFP → Q4, Q6, Q8, IQ3, or IQ4 GGUF

quantbridge convert /models/source ./model-Q4_K_M.gguf --to q4 --llama-cpp /opt/llama.cpp
quantbridge convert /models/source ./model-Q6_K.gguf    --to q6 --llama-cpp /opt/llama.cpp
quantbridge convert /models/source ./model-Q8_0.gguf    --to q8 --llama-cpp /opt/llama.cpp
quantbridge convert /models/source ./model-IQ3_M.gguf   --to iq3 --llama-cpp /opt/llama.cpp
quantbridge convert /models/source ./model-IQ4_XS.gguf  --to iq4 --llama-cpp /opt/llama.cpp

q4, q6, q8, iq3, and iq4 are friendly aliases. Exact names also work:

quantbridge convert /models/source ./model.gguf --to q4_k_s --llama-cpp /opt/llama.cpp

For a quant type added by a newer llama.cpp release:

quantbridge convert /models/source ./model.gguf --to gguf:TYPE --llama-cpp /opt/llama.cpp

The local llama-quantize --help output is checked before the expensive conversion.

FP16 → MXFP4, NVFP4, MXFP8, or FP8

quantbridge convert /models/model-fp16 ./model-mxfp4 --to mxfp4
quantbridge convert /models/model-fp16 ./model-nvfp4 --to nvfp4
quantbridge convert /models/model-fp16 ./model-mxfp8 --to mxfp8
quantbridge convert /models/model-fp16 ./model-fp8   --to fp8
quantbridge convert /models/model-fp16 ./model-fp8   --to nvfp8  # documented FP8 alias

On Apple silicon, auto prefers MLX for MXFP4/NVFP4/MXFP8. Elsewhere it uses AutoRound and writes an LLM-Compressor-compatible checkpoint. Override explicitly with --backend mlx or --backend auto-round.

Choose the AutoRound recipe when needed:

quantbridge convert /models/model-fp16 ./model-nvfp4 --to nvfp4 --quality fast
quantbridge convert /models/model-fp16 ./model-nvfp4 --to nvfp4 --quality best

balanced is the default.

fast selects AutoRound's calibration-free optimized RTN recipe. balanced uses the standard accuracy/time trade-off, while best spends substantially more calibration time for maximum quality.

Inspect or preflight first

quantbridge inspect /models/source --json
quantbridge convert /models/source ./model.gguf --to q4 --check-only --llama-cpp /opt/llama.cpp
quantbridge convert /models/source ./model.gguf --to q4 --dry-run --llama-cpp /opt/llama.cpp

--check-only runs available upstream preflights. --dry-run only prints the route and commands.

Automatic temporary conversion

Most Q/K/IQ types cannot be written directly by the HF-to-GGUF converter. QuantBridge automatically creates an F16 GGUF beside the final output, quantizes it, verifies GGUF magic, and removes the intermediate after success.

If a step fails, the work directory is preserved and printed. This avoids throwing away a large, useful intermediate. Use --work-dir to place it on a drive with more space or --keep-work to retain it after success.

What “universal” means here

QuantBridge is universal at the routing layer: it is not tied to Qwen tensor names, one shard count, one parameter count, or one hard-coded disk estimate. Actual model support is the union of the installed upstream backends.

Source	FP16	GGUF Q/IQ	Microfloat output
Dense local HF Safetensors	Yes, architecture-neutral streaming	If current llama.cpp supports the architecture	Yes
Transformers MXFP4	Mxfp4Config(dequantize=True)	Current llama.cpp direct path	Yes, through dense intermediate when needed
Compressed Tensors MXFP4/MXFP8/NVFP4/FP8	CompressedTensorsConfig(dequantize=True)	Current llama.cpp direct path	Yes, through dense intermediate when needed
ModelOpt checkpoint	When the installed registered loader supports dequantization	When current llama.cpp supports that exact export	Backend-dependent; no unsafe manual fallback
MLX MXFP4/NVFP4/MXFP8	mlx-lm --dequantize	MLX dense intermediate, then llama.cpp validation	Yes
GGUF	No GGUF→HF promise	Yes	No

A model architecture still has to be recognized by the selected backend. “Any model” cannot safely mean “ignore unknown tensor semantics.”

MXFP8 and the nvfp8 name

The published ecosystem uses MXFP8 and NVFP4. There is no corresponding published NVFP8 checkpoint standard. QuantBridge accepts --to nvfp8 as a convenience alias for ordinary NVIDIA-style FP8 (FP8_STATIC) and prints this fact; it does not falsely label that output as a new wire format. Use --to mxfp8 when you need OCP microscaling FP8.

Safety defaults

• Outputs must not exist. There is no force-overwrite option.
• Local dense FP16 conversion validates every Safetensors header before writing.
• The streaming route handles BF16 payloads without relying on NumPy BF16 support.
• FP16 overflow to NaN/Inf stops the conversion.
• Remote model code is disabled unless you explicitly pass --trust-remote-code.
• Commands are executed without a shell.
• llama.cpp preflight is enabled by default.
• Failed large intermediates are preserved for diagnosis.

Important limitations

• Quantization is lossy. Dequantization cannot recover information discarded earlier.
• Direct conversion between two quantized formats still dequantizes internally when the target tool requires it.
• NVFP4 and accelerated MX formats have hardware and kernel requirements defined by their backend.
• IQ and very-low-bit GGUF formats often benefit from a good importance matrix (--imatrix).
• Requantizing an already-quantized GGUF can compound quality loss. It requires --allow-requantize when llama.cpp requires that opt-in.
• Model and tokenizer licenses remain your responsibility.

Tests

python -m unittest discover -s tests -v
python -m compileall -q src

The test suite covers format detection, aliases, command injection rejection, two-stage GGUF planning, overwrite protection, raw BF16 decoding, FP16 dtype verification, and multi-shard index generation.

Upstream contracts

QuantBridge v2.0.0 follows the maintained interfaces documented by:

• Hugging Face FP-Quant and Compressed Tensors for registered checkpoint dequantization;
• MLX-LM convert.py for mxfp4, nvfp4, mxfp8, and dequantization modes;
• AutoRound's current quantization guide for MXFP/NVFP/FP8 schemes and LLM-Compressor export;
• llama.cpp's HF-to-GGUF converter and llama-quantize for architecture mapping and Q/K/IQ output.

These projects evolve independently. Use a current release, run quantbridge doctor, and keep the default preflight enabled.

Publishing

Follow PUBLISHING.md for the exact GitHub commands and release checklist.

License

MIT. See LICENSE. QuantBridge is a community project and is not affiliated with MLX, Hugging Face, NVIDIA, Intel, AutoRound, or llama.cpp.