Kubernetes for Multi-Host Training and Inference: Workload Aware Sc...
summary
Kubernetes adds a workload API (alpha in 1.35) to represent gangs of pods, node topology groups, and dynamic metadata so the scheduler can make placement proposals and accept them atomically; a new scheduling phase schedules multiple pods together to support pod groups and node groups. Use cases: large multi‑host pretraining (hardware failures every few days), reinforcement learning with mixed CPU/GPU work and Ray/Torch Elastic, multi‑pod inference replicas, and research jobs. Design aims for fast recovery (target ~300 pods/sec) and integration with external schedulers.
AI-written summary · about
chapters (10)
- 0:00Multi-host AI workloads intro
- 1:27Pre-training challenges
- 3:12Post-training & RL workloads
- 4:37Serving & inference patterns
- 6:04Research workloads & fairness
- 7:07Scheduling challenges
- 8:12Workload awareness in K8s
- 10:20Scheduling vs awareness
- 13:36Workload API structure
- 18:26Q&A on scheduling
related talks
36mMulti-Cloud Machine Learning Data and Workflow with Kubernetes
31mAdvancements in AI/ML Inference Workloads on Kubernetes From WG Serving and Ecosystem Projects
38mKubernetes for AI Workloads – What Works and What… Doesn’t
16mAI + Kubernetes: What Beginners...
38mBeyond Kubernetes: Adapting To Spe...
40mAI Training & Inference on Kubernetes: Compute Patterns and Best Practices with Karpenter.
36mMulti-Cloud Machine Learning Data and Workflow with Kubernetes
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering