Building Robust Reactive ML Systems In A Multiplayer Setting
summary
Claim: Metaflow provides an event driven pipeline that version controls artifacts, freezes Conda/Pip deps, and auto launches steps on Kubernetes then schedules via Argo so multiple engineers run isolated experiments. Demo shows asynchronous finetuning of LLaMAv2 on fresh data using trigger_on events and Argo orchestration, with runs minted as unique run IDs and namespaced by Metaflow projects namespaces. Built pipelines run end to end but hit GPU capacity limits in the demo.
AI-written summary · about
chapters (11)
- 0:00Introduction to multiplayer machine learning
- 1:14Why ML systems are hard
- 7:54Metaflow for ML challenges
- 9:58Notebooks to Metaflow pipelines
- 13:53Scaling compute with Kubernetes
- 16:25Autonomous deployments with schedulers
- 18:13Event-driven reactive ML workflows
- 20:26Multiplayer ML patterns
- 24:41Training models reactively
- 29:21Live Metaflow demo
- 34:33Metaflow ML infrastructure
related talks
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
44mHow Metaflow Became Netflix's Beloved Data Science Framework
10mLightning Talk: Multiplayer Machine Learning with Metaflow, OpenAI Whisper and...
01h 29mFull-stack Machine Learning and Generative AI for Data Scientists
49mKedro + MLflow - Reproducible and Versioned data pipelines at scale
27mApplied Machine Learning: a Netflix production
40mOptimizing AI/ML Workloads