Why we do ML model retraining wrong, and how to do better
summary
Machine learning models degrade over time so the concrete proposal is: monitor model quality and error distributions, run per-step data checks inside pipelines, detect per-feature data and concept drift, then consult experiments on historical data to find the minimal retrain window (how many days or hours of new labels are actually needed). Build a curated golden set to require new models to beat, consider cheaper fallbacks (rules, older simpler models), automate alerts and retrain actions but keep humans for rare manual reviews.
AI-written summary · about
chapters (17)
- 0:00ML model retraining intro
- 1:50Retraining strategies in production
- 3:36Costs and retraining alternatives
- 6:26Pragmatic model update strategy
- 10:00Detecting model quality drops
- 11:24Data quality testing in pipelines
- 12:50Data and concept drift
- 14:14Sufficient data for retraining
- 15:59Non-ML alternatives
- 17:44Golden set model evaluation
- 19:50Automating retraining decisions
- 20:53Blind retraining cases
- 21:36Testing retraining on historical data
- 22:39Balancing automation and oversight
- 23:00Q&A: Adding new data
- 24:23Q&A: Reinforcement learning for retraining
- 26:10Q&A: Building golden sets
related talks
18mReliable Maintenance of Machine Learning Models
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
39mAn SRE Approach to Monitoring ML in Production
31mReliable Data for Large ML Models: Principles and Practices
29mAutomated Model Update & Evaluation
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
16mMLOps — Understanding and Fighting Model Degradation in Production
38mBuilding robust machine learning systems