Building robust machine learning systems
summary
Systems use concrete practices: automated pipelines with Luigi and BigQuery, offline feature code reused online, model provenance saved as dictionaries and pickled models and S3/Google Cloud Storage, and HTML reports that flag score distribution drift. Labels arrive with long delays (typically 2–3 months label lag), so training drops recent data, uses synthetic worst‑case examples, asserts on critical cases, scores old and new models side‑by‑side in production, and retrains on a 2‑week to 2‑month cadence.
AI-written summary · about
chapters (10)
- 0:00Robust ML systems intro
- 3:55Why ML testing is harder
- 7:25Handling fraud labels
- 12:17Avoiding time leakage
- 14:03Testing data invariants
- 17:57Safe model deployment
- 21:09Gradual rollouts validation
- 23:16Automating ML pipelines
- 26:22Key lessons
- 27:50Q&A
related talks
31mReliable Data for Large ML Models: Principles and Practices
40mProductionizing Machine-Learning Services: Lessons from Google SRE
18mReliable Maintenance of Machine Learning Models
41mBeyond the Algorithm: What Makes Machine Learning Work?
51mKeynote: Fault Tolerant Machine Learning Operations
51mSpeeding up deployment of machine learning models
32mTesting Machine Learning Systems: What it is and why you should care
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based