Automating ML Pipelines With Kubernetes And Airflow
summary
Pipeline moves from manual Docker producers/consumers to an automated system that reads configs from S3, uses Airflow to enqueue one queue per group, and launches Kubernetes Jobs that spin up consumer pods which in turn create per-model worker pods; sequencing of ML steps uses init containers so preprocessing runs before fit, autoscaling uses ASG and nodes spin up on demand, visibility via Prometheus + Grafana and logs in ELK, and a custom Kubernetes multi-job operator handles hundreds of configs reliably.
AI-written summary · about
chapters (18)
- 0:00This is how we used to build models
- 0:42What are we going to talk about?
- 1:14A little bit about us...
- 1:48A little bit about the team...
- 2:15How we used to build models..
- 4:14The problems
- 5:25What is Kubernetes
- 6:46What is Airflow
- 7:34New architecture
- 9:03How it works - Jobs
- 11:17How it really works - Pods
- 11:53How it works even better - Pods
- 12:51Some Kubernetes magic - Auto scaling
- 14:27Scheduling it all with Airflow
- 17:47Usage visibility
- 19:51Error monitoring - ELK stack
- 21:30Learned lessons from automating ML pipelines
- 23:57Questions?
related talks
18mAutomate the boring ML stuff with pipelines
54mScaling Machine Learning Pipelines in Cloud
05mCI/CD Pipelines and Machine Learning
36mMachine learning pipeline Kubeflow, let's start
25mWhen Kubernetes Meets MLflow: Orchestrating MLOps Processes the Cloud Native Way
29mML Engineering Kubeflow
39mDeploy your ML Workloads on Kubernetes with Kubeflow
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines