📚 Advanced NCCL Tutorial: Beyond Collectives
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
NCCL now exposes device-side primitives that let CUDA kernels read/write remote buffers and drive RDMA: symmetric memory avoids per-call registration to cut microsecond latency, GIN GPU-initiated networking offers put+signal RDMA semantics for one-sided transfers, and LSA load-store accessible plus switch-assisted multim reduce/broadcast enable NVLink intranode paths. Device objects: device communicator, window, team, coop/thread scope. Examples and exercises run on DGX H100 and NVL72, code is open source, and a fused MoE kernel reports ~30% speedup versus unfused.
AI-written summary · about
chapters (9)
- 0:00Advanced NCCL intro
- 4:11NCCL device APIs matter
- 13:15NCCL device API basics
- 22:11Load Store Accessible APIs
- 36:32Fusing comms and compute
- 45:24GPU Initiated Networking
- 52:31Symmetric memory for latency
- 1:03:52NCCL Extensions future
- 1:17:00Q&A
related talks
25mPyTorch Symmetric Memory + NCCL Device APIs: A New Path Towards Multi-GP...
15mAutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and...
34mSymmetric Memory in PyTorch: 10x Faster GPU Communication for AI
48mBridging the Gap: Writing Portable C++ Programs for CPU and GPU
19mCollective-on-Ray: High-performance Collective Communication for Distributed Machine Learning on Ray
33mScaling KV Caches for LLMs: How LMCache + NIXL Handle Network and Storage...
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
25mHigh Performance Networking for Distributed DL Training in Production K8s