level: technical
AMD and Meta engineers upstreamed FP8 training support for AMD Instinct GPUs into PyTorch's TorchAO and TorchTitan libraries. The work adds native support for AMD's FNUZ FP8 format, enables grouped GEMM for mixture-of-experts models on ROCm, and introduces a Triton kernel fusion pipeline to reduce quantization overhead. These contributions were merged into the mainline PyTorch repositories, making competitive FP8 performance available out of the box without AMD-specific installations.
On dense models, rowwise FP8 training delivers a 13.4% throughput gain over BF16 on Llama3-8B using 8 MI300X GPUs, with peak memory nearly identical. For MoE architectures like DeepSeek-V3 671B, fused Triton quantization kernels recovered 89% of the FP8 quantization overhead, improving end-to-end throughput by 17% on 8 MI325X GPUs. Individual kernel optimizations achieved up to a 6.2x speedup per MoE layer, reducing time from 7,290 to 1,170 microseconds.
The optimizations target data movement rather than arithmetic, addressing memory-bound quantization steps. They include fusing multi-kernel chains into single Triton kernels, improving memory coalescing, and relaxing atomic memory ordering on AMD hardware. An attempt to expand autotuning search spaces showed no benefit and was reverted, highlighting the need for hardware-aware tuning. Future work includes MXFP8 grouped GEMM kernels for next-generation MI355X GPUs.
why it matters: Teams using AMD Instinct GPUs can now achieve faster FP8 training for large language models without custom libraries, reducing cost and time for AI development.