Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Tra...
summary
Ports Monarch’s runtime and collectives to ROCm so a single-Python controller runs on MI300-class clusters, compiling the same Rust and Python code for CUDA or ROCm and supporting ROCm 7.0+. Uses HIPification with hipify-torch, Rust type aliases, libbyverse/ RDMA fixes and driver API adaptations. Demonstrates fault-tolerant training: LLaMA 38B / 4 TB training with failures injected every 180s and gradient sync every 20 steps, replicas restart locally, rejoin quorums, and global loss still converges on Slurm and Kubernetes.
AI-written summary · about
chapters (6)
- 0:00PyTorch Monarch on AMD GPUs
- 2:30What is Monarch
- 7:28Enabling Monarch on AMD GPUs
- 11:45Fault-tolerant training case study
- 16:44Demo and results
- 19:16Next steps and contributions
related talks
24mMonarch: A Distributed Execution Engine for PyTorch
05mBuild AI Anywhere with ROCm™ Software: AMD and PyTorch Bring...
12mLightning Talk: PyTorch 2.0 on the ROCm Platform
32mPowering Your Generative AI Workloads with AMD and Open-Source ROCm
24mSponsored Session: PyTorch Distributed and Fault Tolerance
03mKeynote: AMD & PyTorch: A Powerful Combination for Generative AI
20mTaming AI Infrastructure Failures with Agentic Debugging
26mEfficient MoE Pre-training at Scale on AMD GPUs With TorchTitan