Scaling Machine Learning Models on Kubernetes
summary
Claims that Kubeflow and related CNCF tools let teams deploy, train and serve models on Kubernetes but pivots to demonstrating a newer operator called Kao (v0.3) that uses a single Kao workspace object to describe a workload, relies on Carpenter GPU provisioner to create right‑sized VM/GPU nodes, pulls models like Microsoft’s fi‑3.5 into a private registry, and reconciles with Carpenter to schedule pods. Demo attempts provisioning on AKS in Azure fail due to missing GPU VM quota, so the end‑to‑end run does not complete.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:14AI model challenges
- 4:06Why deploying LLMs is hard and sustainability concerns
- 7:03MLOps and Kubernetes
- 8:48CNCF cloud-native AI
- 10:59Introducing Kao
- 13:26Kao for AI workloads
- 15:35Kubernetes operator
- 16:39Carpenter vs autoscaler
- 18:46Demo setup
- 20:29Live demo attempt
- 24:20Conclusion
related talks
46mDeploying scalable Machine Learning models with Kubernetes
54mScaling Machine Learning Pipelines in Cloud
51mMachine Learning using Kubeflow and Kubernetes
31mKubernetes for Machine Learning: Managing and Scaling AI Workloads
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
26mKubernetes and MLOps for Scalable and Reproducible Generative AI
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes