Microservices on InfiniBand: 800 Gbps in Distributed Training
summary
Describes using InfiniBand to run microservices for distributed training of a ranking model at Yandex. Focuses on handling 1+ PiB datasets requiring distributed sorted reads, a sharded parameter server with several TB of parameters and per-host throughput around 300+ Gbps (≈37 GB/s), and the operational/business scale of the system.
AI-written summary · about
chapters (9)
- 0:00Welcome
- 5:28GPU hosts and InfiniBand
- 10:46Message routing example
- 12:20PCI affinity and copy optimization
- 17:22Sharding and NUMA nodes
- 19:31Microservices and asynchrony
- 27:52UCX framework features
- 33:41Summary and recommendations
- 44:19Q&A
related talks
19mAI/ML Networking Challenges: The Fast and the Finicky!
42mDifferential Software Solutions on HPC and IBM Power Architecture
22mTrainium and Inferentia
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
06mRDMA Programming Design and Case Studies - For Better Performance Distributed
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
18mScaling Without Limits: KFIN's Journey to Distributed SQL