Demystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
summary
Describes a production ML stack: feature-generation pipelines, a batch model-training service with orchestration, an inference service and an online feedback loop, plus staging and graduated rollouts used for validation. Claims most failures match general distributed-system faults (analysis of 15 years of postmortems: ~40% distributed, ~30% ML-specific). Recommends concrete mitigations: staging environment and monitoring for IO/compute, anomaly checks on feature volumes, model snapshot rollback, pushback and workload prioritization for pipeline backlogs, and modest provisioning slack to recover.
AI-written summary · about
chapters (15)
- 0:00Intro
- 3:494 things you can do for more reliable ML
- 5:02ML on one machine
- 6:32ML in production
- 11:01What makes ML in prod interesting
- 12:37What goes wrong?
- 14:524 things for more reliable ML
- 15:15ML outages from the outside
- 18:14Where changes happen: binaries
- 18:48Where changes happen: configuration
- 19:16Validating binary and config changes
- 20:50Where changes happen: data
- 21:54Validating data updates
- 24:40Improving data integrity
- 25:56Handling pipeline backlogs
related talks
47mMachine Learning in Production: An Intro to MLOps.
21mWhy is Production ML so Hard?
30mChallenges of machine learning: The new ML lifecycle
19mHow ML Breaks: A Decade of Outages for One Large ML Pipeline
01h 05mMachine Learning Models and Productionization
25mBuilding Scalable MLOps Pipelines That Actually Work
41mInfrastructure for ML Applications
52mThe ML Lifecycle in Combat Conditions