source: PyTorch Blog: Introducing TorchTPU

level: technical

google and meta have released torchtpu, a new backend that lets pytorch run natively on google tpus. the project started at google in mid 2025 and is now open source. it aims to fix problems with the older pytorch/xla approach, which required source changes and had slow compile times. torchtpu supports eager execution and torch.compile, so existing pytorch code can move to tpus by changing device strings from cuda to tpu.

the backend supports about 450 aten ops, including forward and backward passes for major deep learning architectures. it has been validated on the 5,000 most downloaded hugging face models, covering llms like llama and qwen, vision models like vit and resnet, and audio models like whisper. torchtpu also works with vllm, sglang, torch titan, and ray for distributed training and inference. however, peak performance may require adjusting model shapes and sharding for tpu hardware.

tpus use bfloat16 for matrix multiplication with fp32 accumulation, which torchtpu maps to pytorch precision modes. the default medium mode is fast and accurate for most workloads, while high and highest modes use multi-pass algorithms that cost 3x and 6x more compute. torchtpu also introduces a defer and fuse eager mode to reduce overhead. the project is used in production by large google cloud customers and is available on github.

why it matters: data scientists can now run pytorch models on tpus without rewriting code, potentially lowering training and inference costs.


source: PyTorch Blog: Introducing TorchTPU