Add MoE GPTQ benchmark script and quantization onboarding docs

[Ti-Tai Wang (titaiwangms)]

Describe your changes

Stacked on top of #2610 (this PR targets b1-gptq-moe, not main).

Adds a manual validation script for comparing perplexity/size before and after quantizing a real
(downloaded) HF checkpoint, plus three onboarding docs under skills/olive/references/ for
quantization work in this repo:

• scripts/quantize_and_compare_perplexity.py — generic, pass-agnostic script (works for any
  registered Olive PyTorch quantization pass, not just GPTQ/MoE) that loads a real model, quantizes
  it, and reports weights-size and WikiText-2 perplexity deltas. This is the same style of
  real-model validation tool that surfaced real RTN bugs in Extend PyTorch RTN weight quantization to MoE experts #2584 after synthetic-model unit tests
  had already passed.
• skills/olive/references/quantization-onboarding.md — general RTN/GPTQ pass onboarding: shared
  config surface, when to use RTN vs. GPTQ, calibration split hygiene.
• skills/olive/references/moe-gptq.md — MoE-GPTQ-specific onboarding: why MoE needs its own
  calibration path, the K-last layout allow-list, the dual fallback-threshold design (Add GPTQ quantization support for K-last MoE architectures #2610), and
  what real-model benchmarking showed about fallback rates and quantization wall-time.
• skills/olive/references/profiling-benchmark-example.md — worked example of running the
  benchmark script and interpreting its output.

Three-model benchmark (bits=4, group_size=128, sym=true, full WikiText-2 train calibration, full test eval)

Model	Baseline PPL	RTN PPL (Δ, time)	GPTQ PPL (Δ, time)	KQuant PPL (Δ, time)	Fallback experts
granite-3.0-1b-a400m-base	6.2877	7.5861 (+1.2984, 8.0s)	6.9560 (+0.6683, 658.7s)	7.5162 (+1.2286, 12.1s)	2/768 (0.3%)
OLMoE-1B-7B-0924	6.6182	7.1091 (+0.4909, 52.6s)	6.8966 (+0.2784, 1499.3s)	7.0507 (+0.4325, 71.8s)	10/1024 (1.0%)
Qwen1.5-MoE-A2.7B	6.4246	6.9251 (+0.5005, 85.8s)	6.6117 (+0.1872, 2475.6s)	6.9318 (+0.5072, 148.2s)	0/1440 (0.0%)

GPTQ consistently beats RTN on perplexity delta across all three models, at a real (but
model-size/expert-count-correlated, not cleanly separable) wall-time cost. See moe-gptq.md for
the full discussion, including the OLMoE layer-2/expert-5 case that empirically validates the
dual fallback-threshold design from #2610.

KQuant (#2618) numbers added for comparison: KQuant is data-free (no calibration set, no
per-expert fallback concept — the "Fallback experts" column doesn't apply to it) and its
quantization time is close to RTN's (both are cheap, uncalibrated passes), but its perplexity
delta tracks RTN's rather than GPTQ's on all three models. All three KQuant runs used
moe=true and forced experts_implementation="eager" at inference (grouped_mm cannot run
against QuantTensor-wrapped experts; see #2619).

Notes

• This PR depends on b1-gptq-moe (Add GPTQ quantization support for K-last MoE architectures #2610): capture_moe_fallback_counts() in the script
  unconditionally imports olive.passes.pytorch.moe_calib, which only exists on that branch.
  Please review/merge Add GPTQ quantization support for K-last MoE architectures #2610 first.
• Went through a full internal review pass (readability/correctness/adversarial/spec-adherence/
  cross-module) before opening; findings incorporated include: fixing pass-name resolution to use
  the actual pass registry (OlivePackageConfig.import_pass_module) instead of guessing module
  paths, several docstring/arithmetic corrections in the reference docs, and hedging a couple of
  causal claims that the 3-data-point benchmark can't fully support.

[Copilot]

Pull request overview

Adds developer-facing onboarding references for PyTorch/Hugging Face weight quantization (RTN/GPTQ, incl. MoE) and introduces a standalone script to manually validate real-checkpoint perplexity/size deltas before vs. after running an Olive quantization pass.

Changes:

• Add scripts/quantize_and_compare_perplexity.py to quantize a real HF model with a chosen Olive PyTorch quantization pass and report WikiText-2 perplexity + size deltas (plus MoE fallback coverage when applicable).
• Add three new reference docs under skills/olive/references/ covering quantization onboarding, MoE GPTQ specifics, and an end-to-end benchmark walkthrough.
• Link the new quantization “deep dive” docs from skills/olive/SKILL.md.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
skills/olive/SKILL.md	Adds pointers to deeper quantization reference docs for contributors.
skills/olive/references/quantization-onboarding.md	General RTN/GPTQ onboarding + shared config surface + where to look in code/tests.
skills/olive/references/moe-gptq.md	MoE GPTQ calibration/fallback design explanation and benchmark-backed guidance.
skills/olive/references/profiling-benchmark-example.md	Worked example of running and interpreting the benchmark script output.
scripts/quantize_and_compare_perplexity.py	New manual validation script for pass-agnostic, real-model perplexity/size comparisons.

Suppressed comments (3)

scripts/quantize_and_compare_perplexity.py:355

• Baseline tokenizer/model loading should pass trust_remote_code through (matching --trust_remote_code and the HfModelHandler load). Without this, the script can fail early on models that Olive could otherwise quantize, and it can also make baseline vs quantized behavior diverge.

    tokenizer = AutoTokenizer.from_pretrained(args.model_id)
    baseline = AutoModelForCausalLM.from_pretrained(args.model_id, **dtype_kwargs).to(args.device)
    if hasattr(baseline, "set_experts_implementation"):

scripts/quantize_and_compare_perplexity.py:369

• HfModelHandler should be constructed with the same trust_remote_code setting as the baseline load. Otherwise, baseline might load successfully (or fail) under one policy while the quantization input model uses another, which defeats the script's stated fairness guarantees.

    input_model = HfModelHandler(model_path=args.model_id, load_kwargs=dtype_kwargs)

scripts/quantize_and_compare_perplexity.py:127

• The return type annotation uses callable, which is a built-in function, not a typing construct. This makes the annotation misleading and can confuse type-checkers/readers. Prefer either a proper Callable[...] annotation (with an import) or drop the return annotation here since this is a CLI helper.

def capture_calibration_dataset() -> tuple[dict, callable]: