When Kubeflow Fights Cilium: Debugging 60% Idle GPUs...
summary
Built a 500-node testbed with Kubeflow and Cilium and find GPUs idle 60% of the time during distributed training because Kubeflow schedules coordinator pods without network topology awareness while Cilium enforces a Cilium deny cross-zone policy that blocks cross-zone NCCL traffic. Controlled experiments raise GPU utilization from 40% to 85% GPU utilization after applying three concrete YAML changes: add nodeAffinity topologySpreadConstraint tolerations so coordinator and workers co-locate in the same zone and network-aware placement no longer stalls training.
AI-written summary · about
chapters (8)
- 0:00Introduction and scenario
- 1:21Defining the problem
- 3:17Live demo of the failure
- 7:44Root cause analysis
- 8:54Topology-aware networking
- 11:44The proposed solution
- 13:24Demonstrating the fix
- 17:09Summary and takeaways
related talks
34mSupercharge Kubeflow Performance on GPU Clusters
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
32mMachine Learning Using Various GPU Technology With Kubeflow.
42mKubeflow From the End User’s Perspective: The Good, The Bad, and The Ugly
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
29mA Scalable Platform for Training and Inference Using Kubeflow
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...