fix: Qwen Image 2.1 alpha output white squares (VAE compute path changed)

[CarlGao4]

Summary

This PR fixes Qwen Image 2.1 resulting images with white squares when writing alpha channels. The cause of this is that VAE weights are forced to be loaded and computed using FP16, while the Qwen Image 2.1 VAE decoder residual stream legitimately reaches magnitudes beyond the f16 range (±65504) in some regions. Then it becomes +inf, which produces NaN in later computation, and be clamped to 255 when converting to uint8.

This changes vae weights loading behavior to keeping original dtype in memory instead of converting to F16 (it also makes vae weights could be affected by --tensor-type-rules, which couldn't before since F16 is hardcoded). The computation would use the same dtype as loaded weights unless a new parameter --vae-dtype is specified (quantized dtypes like q8_0 would be F16). Besides, im2col uses F16 as long as input is not BF16 (which uses F32), so I also added a path to force using F32 when input is F32 for users using vae weights without quantization.

Related Issue / Discussion

This is a solution for the issue left in #2021

fix #2024

Additional Information

It seems that Apple Metal does not support F32 conv, so it would fallback to CPU if F32 computation is used (as long as F32 or BF16 weights are used)

Another concern about Qwen Image 2.1: diffusers prefers LANCZOS input resizing (https://github.com/huggingface/diffusers/blob/e0118ade2f60234c41bacf40330a7e2f61108849/src/diffusers/image_processor.py#L92-L93), while Nearest is forced now; this model also has a default cfg 1.0 according to the diffusers repo (https://github.com/huggingface/diffusers/blob/e0118ade2f60234c41bacf40330a7e2f61108849/src/diffusers/pipelines/qwenimage21/pipeline_qwenimage21.py#L542-L544), while sd.cpp defaults to 7.0 globally.

I suggests that add model-specified default values for cfg scale. I'm still investigating the image resizing path

Checklist

• I have read and confirmed this PR follows the contribution guidelines.

[CarlGao4]

Something off-topic: I've also successfully built SYCL build on Windows, and added standalone CUDA build on Linux. If the community is interested in this I would also open a PR for this

[leejet]

Perhaps we can identify the specific operator involved and force it to use F32 precision. That could fix the issue while keeping the performance impact and VRAM increase minimal.

[CarlGao4]

I think providing an option for full F32 computation is reasonable. Since most weights of vae is F32, force quantizing to F16 is not quite a good idea. When the user uses VAE weights in BF16, we can just keep it and result almost no VRAM increase

[leejet]

Thanks for working on this. I think --vae-dtype is a reasonable feature, but it needs more work on backend compatibility and convolution dispatch before we can merge it.

Convolution support still varies considerably across ggml backends. The im2col + GEMM path can require a large temporary buffer, while some direct convolution paths lack an optimized implicit-GEMM implementation and can be much slower. BF16 support is also incomplete, so changing the default compute type requires checking the entire operation path, including fallbacks.

I found several concrete issues in the current implementation:

1. Default BF16 convolutions fail on CPU. Preserving BF16 weights produces a MUL_MAT(F32, BF16) operation in the im2col path, which the CPU backend does not support.

2. The LTX forced-F32 path can generate an unsupported BF16 im2col output. The encoder's down_blocks.7 enables force_prec_f32, but the im2col output type still comes from w->type. With BF16 weights, the CPU implementation aborts, and the CUDA implementation can write FP32 data into a BF16-sized output buffer.

3. BF16 weights are incompatible with the current direct convolution kernels. With --vae-conv-direct, BF16 weights reach kernels that only accept F16/F32, triggering assertions on CPU/CUDA.

4. F32 weights silently bypass --vae-conv-direct. The F32 branch takes precedence over the direct option for both 2D and 3D convolutions. This forces allocation of the full im2col buffer and can cause OOM in workloads that rely on direct convolution to reduce memory usage.

For the specific Qwen Image 2.1 VAE issue, I think convolution scaling is a better fit. I checked the BF16 VAE file: its weights range from -4.90625 to 9.5, so converting the weights to FP16 does not cause overflow from excessive magnitude. The scaling change targets intermediate activation overflow.

I've added a fixed 1/128 input scale to its VAE convolutions, restoring the output scale before adding bias. This covers encoding and decoding, including the 2D, 3D, and direct paths, while reusing the existing FP16 convolution implementations.

I'd prefer to use that focused fix for this issue and revisit --vae-dtype separately with backend capability checks and appropriate fallbacks.

[CarlGao4]

That's true. I just tested on CUDA using F32 and BF16, which can't cover all circumstances.