Scaleable Distributed Deep Learning
summary
A communications library that plugs into TensorFlow, Caffe and Torch and uses doubled NVLink and a multi-ring topology to keep workers tightly synchronized, achieving 95% scaling efficiency on large runs. Shows fully synchronous training of a ResNet to published top accuracy and beats a prior record (≈66 minutes) by running 256 GPUs across 64 servers (Pascal P100), plus out-of-core large model support using host memory.
AI-written summary · about
chapters (12)
- 0:00Introduction to podcast
- 3:29Guest introduction Hilary Hunter
- 7:43Power AI DDL research overview
- 12:17Power AI and hardware integration
- 18:18Architectural advantages of DDL
- 22:15Developer productivity with DDL
- 25:47Large model support and system resources
- 30:45Impact on deep learning adoption
- 33:32Future of deep learning applications
- 35:38Enterprise AI lifecycle tools
- 38:09Advice for enterprises starting out
- 40:14Closing remarks and Q&A
related talks
49mDistributed Deep Learning
34mDemocratizing Machine Learning on Kubernetes [I]
24mTraining large scale models using PyTorch
25mHigh Performance Networking for Distributed DL Training in Production K8s
08mPyTorch Distributed: Towards Large Scale Training
29mDistributed Deep Learning with Horovod
40mScalable machine-learned model serving
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs