How Metaflow Became Netflix's Beloved Data Science Framework
summary
Metaflow packages data, compute and orchestration so data scientists go from prototype to deployment in days instead of months — two-thirds of production models deploy in a week or less. It uses an immutable artifact store on S3, runs steps on Titus/Kubernetes or AWS Batch, maps to Argo/Step Functions for scheduling, and pins environments via conda dependency control. Adoption hits thousands of projects, over 1 billion steps executed and peak parallelism above 10,000 tasks, with a 2019 open source release and ~2,000 community members.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:49ML production struggles at Netflix
- 3:16ML infrastructure needs
- 5:44ML complexity and platform need
- 7:28Introducing Sonia and her machine learning workflow pain points
- 11:45ML stack and Metaflow concept
- 12:28Metaflow architecture and DAG pipelines
- 14:16Data handling in Metaflow
- 15:17Containerized compute with Metaflow
- 16:44Versioning in Metaflow
- 17:49Deployment models with Metaflow
- 19:12Metaflow as ML infrastructure glue
- 20:15Metaflow's impact on velocity
- 22:02Open-sourcing Metaflow
- 24:08Metaflow team and customer focus
- 26:16Expertise in customer needs
- 29:47Understanding customer pain
- 31:34Pairing with data scientists
- 34:02Teaching best practices
- 36:54Metaflow education and docs
- 38:18Customer obsession success
- 39:44Q&A
related talks
27mApplied Machine Learning: a Netflix production
27mMore Data Science, Less Engineering
24mHuman Friendly, Production Ready Data Science Stack
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
30mHuman-Centric ML Infrastructure at Netflix
35mBuilding Robust Reactive ML Systems In A Multiplayer Setting
36mEnterprise-grade Full Stack ML Platform