Beyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
summary
Cast AI team unifies five heterogeneous clusters using Liqo virtual nodes to make on‑prem A100/H100 and cloud spot GPUs appear as one schedulable topology, integrates Kueue for a global queue and uses node selectors and tolerations to prefer GPU types and spot vs on‑demand. Networking compares Cilium mesh vs Liqo WireGuard, checkpoints and retries handle spot loss, and results show unify 5 heterogeneous clusters, increase GPU utilization 42→78%, jobs 10h→<30m, with new GPUs plug‑in taking 1–3 minutes.
AI-written summary · about
chapters (16)
- 0:00GPU scarcity at Cast AI
- 1:45Underutilized GPUs cross-region issues
- 3:26Adding clusters new issues
- 5:11Unified GPU scheduling needs
- 6:14Lyco connects Kubernetes clusters
- 8:03Virtual nodes multi-cloud GPUs
- 9:25Spot vs on-demand GPUs tolerations
- 10:52Cross-cluster networking Liqo Cilium
- 11:57Queue as global GPU scheduler
- 13:23MultiQ vs Lyco cluster options
- 15:52Failover disaster recovery Lyco
- 16:55Results 78% GPU utilization waits
- 18:42Lessons learned spot pitfalls checkpointing
- 20:07Omni Cast AI open-source GPU backbone
- 21:10Summary virtual nodes Lyco flags
- 21:52Q&A closing
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
38mImproving GPU Utilization using Kubernetes
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
01h 22mWorkshop: Optimizing Kubernetes Cluster Scaling for Adv...
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency
21mUnlocking How To Efficiently, Flexibly, Manage and Schedule Seven AI Chips in Kubernetes
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency
38mBeyond Kubernetes: Adapting To Spe...