Distributed Deep Learning on Apache Mesos with GPUs and Gang Scheduling
summary
Uber runs distributed TensorFlow on Mesos using GPU isolation and nested containers to separate management and user images, adds a custom Peloton scheduler for elastic GPU reservations, locality-aware placement and gang scheduling, and uses MPI + NCCL with a Horovod ring-allreduce implementation. Benchmarks show ImageNet throughput rising from ~5,000 to ~8,000 images/sec and scaling ~30–35× across dozens of GPUs; a real model that took two weeks trains in about seven hours.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:40Deep Learning @ UBER
- 1:59Self-Driving Vehicles
- 2:46Trip Forecasting
- 3:42Fraud Detection
- 4:39Why Distributed Deep Learning?
- 6:09How Distributed Deep Learning Works
- 8:05Why Mesos?
- 9:17Mesos Support for GPUs
- 10:49Mesos Nested Containers
- 12:20What is Missing?
- 15:25Peloton Overview
- 16:22Peloton Architecture
- 18:05Elastic GPU Resource Management
- 18:56Resource Pools
- 19:25Gang Scheduling
- 20:10Placement Strategies
- 21:10Why TensorFlow?
- 23:46Architecture for Distributed TensorFlow on Mesos
- 25:50Can We Do Better?
- 28:41Architecture for Horovod on Mesos
- 29:52Distributed Training Performance with Horovod
- 30:42What About Usability?
- 32:41Giving Back
- 33:13Thank you!
related talks
49mDistributed Deep Learning
34mDeep Learning at Scale
34mDemocratizing Machine Learning on Kubernetes [I]
30mDistributed Deep Neural Network Training using MPI on Python
36mAccelerating High-Performance Machine Learning at Scale i...
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
45mHigh-Performance and Scalable Middleware for HPC and Deep...
38mAll You Need to Know to Build Your GPU Machine Learning Cloud [B]