TorchStore: What We Learned Building Distributed Storage Sol...
summary
TorchStore is an open source distributed tensor store built on Monarch actors that combines save-anywhere semantics with fast direct transfers using pluggable transport backends (RDMA, shared memory, TCP/RPC). It implements transparent live DTensor resharding so producers and consumers can use different parallelism layouts, stages weights into pre-fetching circular volumes to avoid runtime allocation, supports GPU-to-GPU RDMA (0.5s for a 314B cross-node sync reported), and shows correct end-to-end transfers up to 64 nodes in synthetic benchmarks.
AI-written summary · about
chapters (9)
- 0:00TorchStore project intro
- 2:31RL and weight sync basics
- 7:06Sync vs async weight sync
- 11:20TorchStore primitives
- 13:27Performant async RL data flow
- 18:22GPU-to-GPU RDMA for sync RL
- 20:09Resharding challenges
- 23:59Scaling TorchStore
- 25:25Future applications
related talks
24mSponsored Session: PyTorch Distributed and Fault Tolerance
11mLightning Talk: Tensor and 2D Parallelism
24mMonarch: A Distributed Execution Engine for PyTorch
35mSystems @Scale Summer Q&A Week 3
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
08mPyTorch Distributed: Towards Large Scale Training
19mComposable Distributed PT2(D)
23mSponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future