Machine Learning Using Various GPU Technology With Kubeflow.
Talkat KubeCon + CloudNativeCon North America 2022 · on Friday, 2022-10-28· at 16:00· 252 AB, Huntington Place: 1 Washington Blvd·
summary
POC combines NVIDIA A100 MIG up to 7 instances with Kubeflow and finds MIGs give 3–7× speedup on light models when running seven MIG slices simultaneously versus serial runs, but distributed training only works reliably when each process gets a single MIG slice. Platform requires manual YAML/device-plugin tweaks and a dashboard fix to surface MIGs. Separate POC enables GPU Direct RDMA with InfiniBand, delivering 11–62% throughput gain when scaling to 16 GPUs after OS, driver and Kubernetes setup.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:03Kubeflow and Samsung Cloud
- 2:49Multi-Instance GPU intro
- 7:52MIG in Kubeflow POC
- 11:55Distributed training with MIG
- 14:51MIG vs single GPU perf
- 16:40MIG limits for distributed
- 17:01GPU Direct RDMA explained
- 20:13GPU Direct RDMA setup
- 22:28GPU Direct RDMA POC
- 24:19GPU Direct RDMA gains
- 26:08Conclusion and Q&A
related talks
34mSupercharge Kubeflow Performance on GPU Clusters
24mA Better and More Efficient ML Experience for CERN Users
30mFrom High Performance Computing To AI Workloads on Kubernetes: MPI Runtime in Kubeflow TrainJob
51mMachine Learning using Kubeflow and Kubernetes
46mMachine Learning with Kubernetes
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
29mA Scalable Platform for Training and Inference Using Kubeflow
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs