Accelerating Collective Communication in Data Parallel Training across Deep Learning...
Talkat 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22) · 2022-04-04..04-06·
summary
Implements changes to Horovod: response caching that runs the control plane once and uses a global bitvector intersection for readiness, plus explicit all-reduce grouping to force tensorfusion and control buffer sizes. Evaluated on Summit and STEMDL, showing ~2x speedup at 6,000 GPUs, ~12% better than tf.distribute (up to 1,536 GPUs), ~20% better than BytePS at 768 GPUs, and near-linear 0.93 scaling to 27,600 GPUs with sustained 1.54 exaflops FP16 (±0.02).
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:30Deep learning in HPC
- 1:26Horovod overview
- 2:37Control and data planes
- 3:53Coordination workflow
- 6:43Scalability issues
- 8:35Project goals
- 9:23Response caching solution
- 10:45Caching workflow
- 12:51Caching performance
- 13:30All-reduce grouping
- 15:02StemDL performance
- 15:27Summit supercomputer results
related talks
29mDistributed Deep Learning with Horovod
38mDemocratizing Deep Learning at Scale with Horovod
27mDistributed Deep Learning with Horovod on Ray
45mHigh-Performance and Scalable Middleware for HPC and Deep...
26mEfficiently serving LLMs at scale.
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod
26mDeepSpeed: Efficient Training Scalability for Deep Learning
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models