Distributed Deep Learning with Horovod on Ray
summary
Horovod runs data-parallel training for TensorFlow PyTorch and MXNet with near 90% scaling efficiency (demonstrated on IBM Summit with ~27,000 GPUs and a Gordon Bell prize). Ray integration exposes a RayExecutor and a stateful API so jobs launch with gang scheduling, support PyTorch Lightning, and keep model state in Ray’s in‑memory actors. Hyperparameter search uses Ray Tune with a DistributedTrainableCreator. Elastic Horovod provides fault tolerance and auto‑scaling; installable via pip install horovod[ray] and runnable on cloud or Kubernetes.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:10Deep Learning Refresher
- 2:15Distributed Deep Learning
- 3:57Introducing Horovod
- 5:16Horovod Technique: Allreduce
- 6:27Benchmarking: Horovod
- 6:47External Use Case: Oak Ridge National Laboratory
- 7:16Deep Learning at Uber: Recent Trends
- 8:49Challenges using DL at Scale
- 10:42Introducing Ray
- 11:39How Ray Can Help
- 12:40Horovod on Ray: Executor Setup
- 14:09Horovod on Ray: Stateless API
- 15:21Horovod on Ray: Stateful API
- 17:33Horovod on Ray: Hyperparameter Search with Ray Tune
- 18:53What's next?
- 20:01Elastic Horovod: Control Flow
- 20:52Elastic Horovod. Hypersched
- 22:02Ludwig: End-to-End DL without Code
- 23:09Scaling Ludwig
- 24:27Unified Ray Backend
- 25:12Recap
related talks
38mDemocratizing Deep Learning at Scale with Horovod
26mA Growing Ecosystem of Scalable ML Libraries on Ray
29mDistributed Deep Learning with Horovod
33mOvercoming Distributed ML Challenges with Ray Train
22mLarge Scale Data Loading and Data Preprocessing with Ray
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
21mEnd-to-End AutoML with Ludwig on Ray
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod