source: hugging face blog: bringing nunchaku 4-bit diffusion inference to diffusers
level: technical
large diffusion transformers need 20-30 gb of vram in bf16, limiting them to high-end gpus. quantization reduces memory, but most diffusers backends are weight-only, saving memory without speeding up inference. svdquant, the method behind nunchaku, uses 4-bit weights and activations to cut memory and accelerate the denoising loop. previously, using these checkpoints required a separate library. now, diffusers loads nunchaku checkpoints with from_pretrained(), with no local cuda compilation needed thanks to prebuilt kernels from the hub.
nunchaku lite patches standard diffusers linear layers with runtime svdq and awq layers. svdq_w4a4 handles attention and mlp projections with 4-bit weights and activations plus a low-rank correction, available in int4 and nvfp4. awq_w4a16 uses 4-bit weights with 16-bit activations for memory-bound normalization layers. this approach delivers about 30% speedup and up to 50% vram reduction compared to bf16. combining with torch.compile can boost speedup to 1.8x, and quantizing text encoders with bitsandbytes nf4 further reduces memory.
the diffuse-compressor toolkit lets users quantize their own models. it scans the architecture, applies svdquant to transformer blocks, and packages the result as a standard diffusers repository with a quantization_config. for example, quantizing flux.2 klein 4b involves inspecting targets, running quantization, and converting to a pipeline. some architectures need structural rewrites for fused operations, which require model-specific configs. nvfp4 checkpoints need nvidia blackwell gpus, while int4 works on turing, ampere, and ada gpus.
why it matters: this makes high-quality image generation accessible on consumer gpus by reducing memory and latency, simplifying deployment for ai practitioners.
source: hugging face blog: bringing nunchaku 4-bit diffusion inference to diffusers