Distributed Deep Learning with Horovod
summary
Horovod is a library that runs on top of TensorFlow and PyTorch and uses MPI plus NVIDIA’s NCCL for fast collectives, implementing ring-allreduce and a hierarchical reduce variant; it pins one process per GPU, broadcasts initial weights from rank 0, and writes checkpoints only from that rank. It batches small gradients with tensor fusion, supports FP16 allreduce, integrates with Spark and Petastorm, and demonstrates about 90% scaling efficiency to 512 GPUs for ResNet/Inception (≈70% for VGG).
AI-written summary · about
chapters (8)
- 0:00Start
- 5:04Data Parallel Training
- 9:49Worker Coordination
- 13:26Checkpointing
- 17:44Data Preparation
- 20:00Tensor Fusion
- 21:22Mixed Precision Training
- 24:31Gradient Compression
related talks
38mDemocratizing Deep Learning at Scale with Horovod
27mDistributed Deep Learning with Horovod on Ray
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
30mDistributed Deep Neural Network Training using MPI on Python
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform
49mDistributed Deep Learning
45mHigh-Performance and Scalable Middleware for HPC and Deep...