source: pytorch blog: pytorch 2.13 release blog
level: technical
pytorch 2.13 adds flexattention support on apple silicon with hand-written metal kernels, giving up to 12x speedup over sdpa on sparse attention patterns. the release also includes a deterministic backward path for flexattention on cuda, ensuring reproducible gradients with negligible overhead. a broad set of mps operations now use native metal kernels instead of mpsgraph, reducing per-op compilation cost and launch latency.
a new nn.linearcrossentropyloss module fuses the final linear projection and cross-entropy computation, cutting peak gpu memory by up to 4x for large-vocabulary language models. it supports label smoothing, weight tying, and z-loss, and works with torch.compile. torch.load now natively handles safetensors files, removing the need for a separate library. python 3.15 wheels are available for linux, including free-threaded builds, though torch.compile is not yet supported.
for distributed training, the new torchcomms backend improves fault tolerance, scalability, and debuggability over the existing c10d backends. fsdp2 can now overlap reduce-scatter and all-gather communications using a separate process group, increasing throughput. the inductor compiler gains a cutedsl backend for high-performance gemm and normalization kernels, and torch.compiler.set_default_backend simplifies using custom backends. platform updates include rocm aotriton 0.12b, armv9-a target support, and intel xpu device telemetry apis.
why it matters: these updates make pytorch faster and more memory-efficient for training large models, especially on apple hardware and in distributed settings, while simplifying model loading and custom compiler backends.