Improving Machine Learning Development Reliability
summary
Meta builds a production model service that catalogs every model, shows owners and on‑call rotations, and serves thousands of users monthly; training backlog shrinks without buying hardware after finding that >50% servers use <5% GPU/CPU and adding role‑aware diagnostics, auto‑tuning and job packing; an automated stopper ml Guardian kills stalled or low‑quality runs; pipeline runs ~10,000 exploration jobs/day, thousands of A/B tests and only a few hundred become production, exposing reproducibility and multi‑system SLA multiplication issues.
AI-written summary · about
chapters (10)
- 0:00Start
- 8:18Software Development LifeCycle
- 8:49ML Development LifeCycle
- 14:30Ownership Challenges
- 15:53Ownership Solutions
- 23:11Capacity Crunch Debugging
- 28:37ML Guardian
- 30:08Admission Control Evolution
- 33:03Reliability
- 36:12AutoML Package Release
related talks
25mBuilding Scalable MLOps Pipelines That Actually Work
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
01h 05mMachine Learning Models and Productionization
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
51mSpeeding up deployment of machine learning models
32mAutomating Production Production Level Machine Learning on AWS
10mFrom Model to Production: Building Scalable MLOps Pipelines
39mAn SRE Approach to Monitoring ML in Production