Reliable Maintenance of Machine Learning Models
summary
Explains that deployed models need ongoing checks of both software metrics (latency, uptime) and statistical metrics (accuracy, calibration), demonstrates detecting data drift by comparing input distributions and concept drift by tracking changing input–output relationships, and shows how real-world examples (laundry customer predictions, streaming recommendations, Stripe fraud blocking) create feedback loops that break retraining. Recommends reproducible scripts, dashboards and vetiver (R/Python) for moving from manual checks to automated monitoring and alerting.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:28Performance
- 6:04Vocabulary
- 9:51Feedback loops
- 12:33Model monitoring maturity
- 14:40Questions
related talks
39mAn SRE Approach to Monitoring ML in Production
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
28mWhy we do ML model retraining wrong, and how to do better
16mMLOps — Understanding and Fighting Model Degradation in Production
32mReliable Machine Learning
19mModel Performance Monitoring - A Practitioner's Perspective
30mHow to Properly Test ML Models & Data
34mTraining & Monitoring AI - Drift Detection