Applied Machine Learning: a Netflix production
summary
Netflix describes Metaflow as a Python-first workflow layer that actually ships: it provides a DAG decorator API, resume-any-step reproducibility backed by an immutable data artifact store, direct S3 partition reads that reach ~10 Gbps, a Titus plugin for requesting CPUs/GPUs and automatic horizontal scaling with for_each/join. Adoption metrics: 134 projects use it internally and about 40% use Titus, with roughly a quarter of projects running in production.
AI-written summary · about
chapters (8)
- 0:00The data science challenge
- 1:27Overcoming production hurdles
- 5:43Introducing Metaflow
- 7:00Designing for human diversity
- 11:15Development and reproducibility
- 14:38Scaling and data access
- 18:55Deployment and monitoring
- 21:50Future work and Q&A
related talks
44mHow Metaflow Became Netflix's Beloved Data Science Framework
27mMore Data Science, Less Engineering
30mHuman-Centric ML Infrastructure at Netflix
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
29mEvolving the ML Platform Organisation at Netflix: A Case Study
35mBuilding Robust Reactive ML Systems In A Multiplayer Setting
04mInterview