Scaling ML Pipelines with Feast, Ray and Kubeflow
summary
Built a file-based pipeline into a distributed stack using Feast for feature management, Ray as the batch engine, and Kubeflow Training Operator to run PyTorch DDP across multiple GPUs. Materialization runs as Ray jobs that populate Feast’s offline and online stores, the same repo code scales from laptop to cluster via a YAML config. A Temporal Fusion Transformer trains on 421K Walmart rows, cuts training time noticeably and hits 10.5% MAPE versus 15–20% baseline, with production inference served from Feast.
AI-written summary · about
chapters (9)
- 0:00MLOps introduction
- 2:23MLOps pipeline challenges
- 6:24Feature engineering importance
- 17:56Feature store and its use
- 25:24Distributed feature engineering
- 30:48Feast and Ray integration
- 38:02Distributed model training
- 42:55Model serving with KServe
- 44:40Summary
related talks
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
33mOvercoming Distributed ML Challenges with Ray Train
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
29mScaling ML Pipelines for Autonomous Driving with Ray
42mDeep learning beyond the learning
31mTransforming Data Science at PepsiCo: The Kubeflow Revolution
32mAn Overview of CloudKitchens's Ray-Powered ML Platform
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...