Skip to content

feat: add PixArt model family support - #2047

Merged
leejet merged 2 commits into
leejet:masterfrom
losewayy:feat-pixart
Sep 25, 2026
Merged

leejet merged 2 commits into
leejet:masterfrom
losewayy:feat-pixart

Conversation

@losewayy

Copy link
Copy Markdown
Contributor

Summary

Closes #1965.

Add support for the PixArt-Sigma transformer (DiT architecture with adaLN-single modulation):

  • New runner src/model/diffusion/pixart.hpp: patch embed via strided conv, 2D sincos position embedding with multi-resolution coordinate scaling, 28× adaLN-single blocks (self-attn + T5 cross-attn + ffn), learned-variance output (8ch → 4ch slice). Config is detected from weight names/shapes rather than hardcoded.
  • Detection: PixArt shares the adaln_single.emb.timestep_embedder signature with LTX-AV; disambiguated by pos_embed.proj which is PixArt-only.
  • Sigma schedule: PixArt trains with linear beta (0.0001 → 0.02), added calculate_alphas_cumprod_linear_beta used when version is PixArt.
  • VAE: PixArt uses the standard SD1.x autoencoder (z=4, scale_factor=0.18215) — excluded from the DiT VAE branch and added to the SD1/SD2 scale-factor branch.
  • T5 tokenizer fix: empty input no longer emits a stray ▁ piece; uncond conditioning now produces <eos> + padding, matching Hugging Face. This affects all T5-based models' negative-prompt path — happy to split into a separate PR if preferred.
  • Quantization: scale_shift_table is excluded from conversion (element-wise views are invalid on quantized block layouts, and modulation params shouldn't lose precision anyway — same latent issue exists in LTXV). A defensive ggml_cast dequantizes the table if a pre-quantized file is loaded.

Tested with PixArt-Sigma-XL-2-1024-MS (standalone diffusers-format transformer + T5-XXL + SD VAE):

sd-cli --diffusion-model transformer.safetensors --t5xxl t5xxl.safetensors --vae vae.safetensors -W 1024 -H 1024 ...

Test plan

  • 1024×1024 generation matches diffusers PixArtSigmaPipeline reference (same composition, photo quality)
  • Single-forward numerical parity vs diffusers: max diff ≤ 0.005
  • T5 embeddings (cond + uncond) parity vs HF: max diff ≤ 0.008
  • q8_0 conversion + inference; legacy fully-quantized gguf also runs via defensive dequant
  • --diffusion-fa path OK
  • SD1.5 regression: end-to-end generation unchanged
  • 512-MS checkpoint (arch identical, needs --model-args for pos-embed base size — documented)
  • Other backends/OS: covered by CI

losewayy and others added 2 commits September 25, 2026 02:19
Add PixArt-Sigma DiT support: standalone diffusers-format checkpoints
are detected via pos_embed.proj and routed to a new PixArtRunner
implementing adaLN-single modulation, 2D sincos position embedding and
learned-variance (8ch -> 4ch) output slicing.

- linear beta schedule (0.0001 -> 0.02) for the compvis denoiser
- SD1.x VAE latent config (z=4, scale_factor=0.18215)
- keep scale_shift_table in f32 (element-wise views are invalid on
  quantized layouts); dequantize defensively when already quantized
- t5 unigram tokenizer: emit no pieces for empty input, matching HF
  so uncond conditioning is <eos> + padding
@leejet
leejet merged commit 39ada08 into leejet:master Sep 25, 2026
7 of 9 checks passed
@leejet

leejet commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Thank you for your contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] pixart model family

2 participants