source: pytorch blog: bringing pytorch monarch to amd gpus: single-controller distributed training on rocm
level: technical
training large language models across hundreds of gpus faces frequent hardware failures. traditional checkpointing wastes computation and leaves clusters idle during restarts. pytorch monarch offers a different approach: a single python program controls the entire cluster, using actors and supervision trees to isolate crashes and restart only failed parts. this keeps healthy nodes working while recovery happens in seconds.
porting monarch to amd gpus required adapting the gpu runtime and communication stack to rocm. the team used hipify_torch to convert cuda code to hip, linked against rccl for collectives, and added a rust compatibility layer so the core code stays platform-agnostic. all 1,171 tests pass on rocm 7.0+. the system now runs on slurm, kubernetes, and skypilot, supporting torchft for fault tolerance and torchtitan as the training engine.
in tests on amd mi300 and mi355 clusters, monarch kept training through injected failures. on a 128-gpu slurm cluster, workers fluctuated but loss converged smoothly without full restarts. on a 256-gpu kubernetes cluster, recovery was stable with minimal participant drop. when a node fails, a peer checkpoint transfer syncs the recovering replica from a healthy one, and training resumes with a new quorum. this avoids global checkpoint reloads and manual intervention.
why it matters: this integration makes large-scale ai training on amd gpus more reliable and efficient by reducing wasted compute from hardware failures.
source: pytorch blog: bringing pytorch monarch to amd gpus: single-controller distributed training on rocm