Symmetric Memory in PyTorch: 10x Faster GPU Communication for AI
summary
Introduces symmetric memory for zero‑copy RDMA in PyTorch using NCCL 2.29 one‑sided APIs, claiming up to 10× lower latency for GPU communication and sub‑10 ms per operation for small messages. Direct GPU pointers share the same virtual address across GPUs via NVLink, avoiding host copies; primitives put_signal, wait_signal, barrier implement one‑sided transfers and synchronization. Integration with torch.compile via a registration API lets operators request symmetric buffers; live demos report ~35% throughput lift for tensor‑parallel LLM inference.
AI-written summary · about
chapters (19)
- 0:00Symmetric memory in PyTorch
- 1:02GPU communication bottleneck
- 3:56Overhead of collective ops
- 5:17Symmetric memory solution
- 7:24What makes memory symmetric
- 9:13PyTorch workflow with symmetric memory
- 10:37Device-side view on symmetric memory
- 11:48Core primitives of symmetric memory
- 14:17One-sided ops unlock flexibility
- 14:40NCCL 2.29 portability contribution
- 16:48torch.compile integration contribution
- 20:23NCCL device APIs infrastructure
- 23:59Layered architecture of symmetric memory
- 24:21NCCL EP for MoE models
- 26:10Pre-built symmetric memory collectives
- 28:20torch.compile integration details
- 30:05Performance on GB200 systems
- 31:53Symmetric memory vs NCCL
- 32:57Q&A
related talks
25mPyTorch Symmetric Memory + NCCL Device APIs: A New Path Towards Multi-GP...
19mSymmetric Memory: A New Paradigm for Programming Distributed AI
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
01h 17m📚 Advanced NCCL Tutorial: Beyond Collectives
15mColocating ML Inference and Training with Fast GPU Memory Handover
24mSponsored Session: PyTorch Distributed and Fault Tolerance
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
19mAI/ML Networking Challenges: The Fast and the Finicky!