Large Scale Distributed Deep Learning on Kubernetes Clusters
summary
Compares deployment of distributed TensorFlow on Kubernetes using tf-operator vs mpi-operator, measuring how different distribution strategies (parameter-server, MirroredStrategy, MPI all‑reduce) affect CPU/GPU/network utilization and idle GPU time; finds parameter‑server causes heavy cross‑host network traffic and GPU stalls while MPI all‑reduce reduces network hotspots; recommends optimizing TF data I/O pipeline so GPUs stay fed; notes TF 1.14/2.0 shifts (eager execution, tf.keras) and that both operators share similar CRD specs and lifecycle management.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:12Speakers
- 1:12TensorFlow 2.0 Workflow
- 5:53Orchestration for DL
- 12:23AllReduce
- 14:02Parameter Server
- 16:46Kubernetes Operators
- 22:47Mirror Strategy in TF
- 23:29TensorFlow + Hovorod
- 25:14PyTorch + Hovorod
- 25:50Recall: TFJob vs. MPIJob
- 30:22Shared API and Best Practices
related talks
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
29mLarge Scale Distributed Deep Learning with Kubernetes Operators
37mBuilding Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I]
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
34mDemocratizing Machine Learning on Kubernetes [I]
27mHarnessing Kubernetes for Scalable AI/ML Workloads
46mMachine Learning with Kubernetes
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster