Monarch: A Distributed Execution Engine for PyTorch
summary
Monarch provides a single-controller Python API that treats a cluster like one process, exposing hosts/processes/actors as typed meshes and supporting scalable broadcast/aggregate messaging. It adds RDMA zero-copy handles for one-sided GPU/CPU transfers that hit ~90–95% of nickel throughput, plus a supervision hierarchy for progressive fault recovery. Benchmarks include Quen 2.57B scaled from 16 to 248 H200s with stable convergence, and a 30-node 240‑GPU Quen-33 (32B) run with 100 injected faults showing ~90s process recovery and ~2.5 min machine recovery.
AI-written summary · about
chapters (16)
- 0:00Monarch introduction
- 1:45Python API for distributed ML
- 3:30Hello world Monarch code
- 5:16Scaling Monarch locally to cluster
- 6:41Optimized messaging for meshes
- 8:28RL training systems with Monarch
- 9:52Fault tolerance via supervision trees
- 12:07Point-to-point RDMA transfer
- 14:55Single controller tensors
- 16:43Monarch examples and benefits
- 17:25RDMA performance and use cases
- 18:29Integration with Verl for RL
- 19:11Torch Forge on Monarch
- 20:57Monarch and TorchFT for training
- 22:23Developer experience with Lightning AI
- 23:27How to try Monarch
related talks
12mMonarch: An API To Your Supercomputer
20mBringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Tra...
26mTorchStore: What We Learned Building Distributed Storage Sol...
22mMonarch: A Fuzzing Framework for Distributed File Systems
25mHigh Performance Networking for Distributed DL Training in Production K8s
34mMonarch, Google’s Planet Scale Monitoring Infrastructure
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads