Supercharge Kubeflow Performance on GPU Clusters
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 17:20· Room 11AB - San Diego Convention Center, Upper Level·
summary
Lab tests on a Cisco UCS 480 ML with eight GPUs use a ResNet TF benchmark and show image throughput rising as GPUs scale and falling by ~40–50% without NVLink; six NVLink lanes give ~300 GB/s vs ~30 GB/s for PCIe (≈10×). Examples use MPI job and Kubeflow operators to launch launcher + workers, request GPUs, and run distributed training and Katib hyperparameter sweeps; demos show scaling by adjusting worker replicas and slots per worker.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:38Agenda
- 1:22What is the Opportunity?
- 2:21Scaling with NVLink
- 7:36Kubeflow Operators
- 9:22Deep Neural Network
- 11:17Data Parallelism
- 13:44Distributed Machine Learning (ML)
- 16:31Distributed Training
- 17:00Sync Parameter Server
- 19:00MPI Job Example
- 19:56MPI Job Spec - Worker
- 20:17MPI Job Scaling
- 20:28MPI Job Spec Scale Out
- 20:54MPI Job Spec - Scale Up
- 22:45MPI Job Execution
- 24:04Multi-Model Parallelism
- 24:30Hyperparameter Tuning
- 25:12Katib Experiment Spec
- 26:31Kubeflow Experiment Execution
- 27:10Infrastructure and OS Optimization
- 27:29Kubernetes GPU Configuration
- 30:03Intra-Node GPU Topology
- 32:11Summary
- 33:33Kubeflow Contributions
related talks
32mMachine Learning Using Various GPU Technology With Kubeflow.
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster
29mA Scalable Platform for Training and Inference Using Kubeflow
25mHigh Performance Networking for Distributed DL Training in Production K8s
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
30mFrom High Performance Computing To AI Workloads on Kubernetes: MPI Runtime in Kubeflow TrainJob
24mA Better and More Efficient ML Experience for CERN Users
22mWhen Kubeflow Fights Cilium: Debugging 60% Idle GPUs...