PyTorch Distributed RPC
summary
Covers the PyTorch RPC package with concrete APIs: rpc.sync, rpc.async, remote for running callables remotely, remote references for zero-copy pointers between processes, and a distributed autograd context that stores gradients in per-backward contexts. Shows a coordinator example wiring workers (w0→w1,w2,w3), explains how remote refs increment refs and fetch data on demand, and describes a distributed optimizer that pulls gradients from the context to update owners. Supports parameter-server, sharded parameters, model-parallel partitioning, and pipeline parallelism; tutorials and API docs exist and examples run end-to-end.
AI-written summary · about
chapters (7)
- 0:00Introduction to PyTorch RPC
- 0:37Why use PyTorch RPC
- 2:37Core RPC features
- 3:39Remote execution APIs
- 5:21Using remote references
- 6:56Distributed autograd
- 8:58RPC use cases and conclusion
related talks
08mPyTorch Distributed: Towards Large Scale Training
19mComposable Distributed PT2(D)
24mSponsored Session: PyTorch Distributed and Fault Tolerance
24mTraining large scale models using PyTorch
20mBringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Tra...
23mSponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future
26mTorchStore: What We Learned Building Distributed Storage Sol...
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads