Large Scale Distributed Deep Learning on Kubernetes Clusters
summary
Compares tf-operator and mpi-operator on Kubernetes for large-scale TensorFlow training, showing they use different distribution strategies: parameter server versus all-reduce/MPI. Demonstrates that network-heavy, multi-host training often makes MPI/all‑reduce more efficient than parameter servers, and that poorly pipelined I/O leaves GPUs idle. Describes using TF Keras/TFFunction, Kubernetes operators and Kubeflow, and argues orchestration and input pipeline tuning, not just GPUs, drive cost and utilization improvements.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:12Speakers
- 1:12TensorFlow 2.0 Workflow
- 5:53Orchestration for DL
- 8:28Parameter Server
- 11:40Reduce
- 16:46Kubernetes Operators
- 22:47Mirror Strategy in TF
- 23:29TensorFlow + Hovorod
- 25:14PyTorch + Hovorod
- 25:50Recall: TFJob vs. MPIJob
- 30:22Shared API and Best Practices
related talks
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
29mLarge Scale Distributed Deep Learning with Kubernetes Operators
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
34mDemocratizing Machine Learning on Kubernetes [I]
46mMachine Learning with Kubernetes
30mFrom High Performance Computing To AI Workloads on Kubernetes: MPI Runtime in Kubeflow TrainJob
27mHarnessing Kubernetes for Scalable AI/ML Workloads
37mBuilding Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I]