Monarch: An API To Your Supercomputer
summary
Monarch presents a Pythonic API that programs GPU clusters as meshes of hosts, processes and actors, exposing precise placement, lifecycle and one-sided memory I/O. It uses RDMA buffers and FUSE-mounted RDMA to mount local files everywhere, implements scalable multicast and accumulation, and runs a demo orchestration where an agent (Claude) fixes a missing tqdm, patches a CUDA kernel causing NaN losses, resolves an off-by-one deadlock, and boots distributed training across four 8‑GPU nodes. Also shows a serverless SPMD flow and agentic legibility.
AI-written summary · about
chapters (7)
- 0:00Monarch project intro
- 1:23Monarch API concepts
- 3:29RDMA buffers for data movement
- 4:53Training app with Monarch
- 6:17Agentic legibility with Monarch
- 6:37Serverless SPMD demo
- 10:47Scaling GPT-2 training
related talks
24mMonarch: A Distributed Execution Engine for PyTorch
34mMonarch, Google’s Planet Scale Monitoring Infrastructure
22mMonarch: A Fuzzing Framework for Distributed File Systems
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
20mScheduler and Sharding Considerations for Network Efficiency
19mSymmetric Memory: A New Paradigm for Programming Distributed AI
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
08mWelcome Back