Automated Model Update & Evaluation
summary
Breaks down when to use stateful (incremental) vs stateless retraining and reports a real example: switching to incremental retraining cuts training cost ~45x and raises purchase-through rate ~20%. Describes pipelines using streaming systems (real-time features and label joins) and CI/CD for models with a model store that packages, validates, shadow-tests and deploys models. Flags two hard limits: fresh labeled data and robust online evaluation (A/B can take weeks for rare classes).
AI-written summary · about
chapters (10)
- 0:00Automated model updates
- 1:29Continuous learning importance
- 4:00Stateful vs stateless training
- 6:48Continual learning stages
- 10:24Pipeline bottlenecks
- 12:48CI/CD for ML
- 17:04Monitoring unlabeled models
- 21:17Detecting data shifts
- 25:12Monitoring predictions & features
- 28:23Q&A
related talks
40mModel Upgrade Schemes- Considerations for Updating Production Models
38mBuilding robust machine learning systems
28mWhy we do ML model retraining wrong, and how to do better
59mContinuous Intelligence: Keeping your AI Application in Production
27mModel Evaluation for Humans
18mReliable Maintenance of Machine Learning Models
40mProductionizing Machine-Learning Services: Lessons from Google SRE
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based