Productionizing Machine Learning Models at Scale with Kubernetes
summary
Builds a Kubernetes-based pipeline at 2U that runs roughly a thousand training and inference processes per day and spikes to tens of thousands in a week, using Kubernetes jobs and cronjobs for batch work, shared Docker base images for fast onboarding, and small Flask/Plumber REST services for online inference. Results: massive parallelism works but cluster quota and lack of auto-scaling cause resource exhaustion, requiring manual node adds, metadata cleanup jobs, and recommendations to add Airflow, Argo or a scheduler.
AI-written summary · about
chapters (10)
- 0:00Introduction to 2u and ML
- 1:00Company overview and use cases
- 2:45Machine learning operations
- 5:00Scaling challenges at 2u
- 8:15Adopting Kubernetes
- 9:45Kubernetes architecture
- 12:48Production workflows
- 15:30Successes and optimizations
- 18:51Lessons and operational bugs
- 23:13Future improvements and advice
related talks
46mDeploying scalable Machine Learning models with Kubernetes
54mScaling Machine Learning Pipelines in Cloud
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
26mKubernetes and MLOps for Scalable and Reproducible Generative AI
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
51mMachine Learning using Kubeflow and Kubernetes