Sponsored Session: PyTorch Distributed and Fault Tolerance
summary
Open source torchcoms provides a thin, object oriented communicator abstraction over new backends like nickel x and Catran transport, claiming fast startup and scale tested on Llama 3 (24,000 GPUs) and Llama 4 (100,000+ GPUs). It exposes GPU-resident collectives and GPU‑initiated collectives to avoid CPU roundtrips, a window/transport API for RDMA and one‑sided puts, pip‑installable third‑party backends for no‑recompile integration, and a fault‑tolerant backend called Mikkel that supports graceful shrink/reconfigure recovery.
AI-written summary · about
chapters (11)
- 0:00TorchComs and PyTorch Distributed intro
- 1:05Motivation for TorchComs
- 3:36Core API design
- 6:25Catran transport layer
- 8:31GPU-resident collectives
- 11:40New APIs and fault tolerance
- 13:48Third-party backend support
- 16:40Fault tolerance strategies
- 18:46Mikkel backend
- 20:30TorchFT improvements
- 21:57Closing and Q&A
related talks
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
19mComposable Distributed PT2(D)
20mBringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Tra...
08mPyTorch Distributed: Towards Large Scale Training
26mTorchStore: What We Learned Building Distributed Storage Sol...
24mTraining large scale models using PyTorch
25mHigh Performance Networking for Distributed DL Training in Production K8s
19mSymmetric Memory: A New Paradigm for Programming Distributed AI