source: hugging face blog: profiling in pytorch (part 3): attention is all you profile
level: technical
this post profiles attention, a core transformer operation, starting with a naive implementation of causal attention built from primitives like matmul, scaling, masking, and softmax. the profiler trace shows five expected gpu kernels plus an unexpected memory copy caused by an out-of-place masked_fill. switching to an in-place masked_fill_ removes the memcpy, saving one kernel per forward pass. this small change adds up across many layers in large models.
pytorch's scaled dot product attention (sdpa) wraps attention into a single function that dispatches to different backends. the math backend, a reference implementation, launches 20 kernels per forward—much slower than the naive version—because it upcasts to fp32, rebuilds the causal mask each call, and uses a safe softmax to avoid nans. the efficient backend fuses everything into one fmha_cutlassf kernel that runs in bf16 on tensor cores, while the flash backend similarly uses a single fused kernel optimized for memory efficiency.
the profiler reveals how backend choice drastically changes kernel count and hardware usage. the math backend runs on slower cuda cores, while efficient and flash backends leverage tensor cores and keep data in fast register memory. understanding these traces helps developers pick the right backend and avoid hidden costs like unnecessary memory copies or mask materialization.
why it matters: profiling attention implementations helps data scientists and ml engineers choose the fastest backend for their hardware, reducing training and inference time for transformer models.
source: hugging face blog: profiling in pytorch (part 3): attention is all you profile