Distributed Deep Neural Network Training using MPI on Python
summary
Hands-on description of distributing DNN training in Python using TensorFlow + Keras for model code, MPI allreduce to sync gradients across workers, and MVAPICH2-optimized MPI for CPU/GPU networks. Horovod is noted as a high‑level wrapper (Horovod uses MPI backend). A tiny Keras example is shown, and empirical scaling data appears: ~500 images/sec on one GPU, ~60,000 images/sec on 512 GPUs (ideal ~70,000, observed ~65,000 due to communication overhead). Continuous-gradient averaging works but communication limits perfect scaling.
AI-written summary · about
chapters (15)
- 0:00Introduction and overview
- 1:00Deep learning fundamentals
- 3:31History and modern trends
- 4:38Deep learning frameworks
- 7:02Advantages of using Python
- 8:43Parallelization necessity
- 9:47Message passing interface
- 11:57Communication patterns
- 13:38Neural network training
- 17:14Implementing models in code
- 18:33TensorFlow playground demo
- 20:43Distributed training strategy
- 23:26Performance at scale
- 24:37Conclusion and resources
- 26:18Question and answer session
related talks
29mDistributed Deep Learning with Horovod
34mDemocratizing Machine Learning on Kubernetes [I]
45mHigh-Performance and Scalable Middleware for HPC and Deep...
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform
34mDeep Learning at Scale
38mDemocratizing Deep Learning at Scale with Horovod
49mDistributed Deep Learning
37mDistributed Deep Learning on Apache Mesos with GPUs and Gang Scheduling