Detecting drift: how to evaluate and explore data drift in machine learning systems
summary
Uses drift as an early proxy for performance when ground truth is unavailable and gives concrete methods: choose parametric tests (t/z) for sensitive detection or non‑parametric tests (KS/Anderson, chi‑square) and distances like Wasserstein, apply sampling or bucketing for huge datasets, use rolling windows for streams, and tune thresholds by replaying historical periods. Evidently produces HTML/JSON dashboards and can auto‑select tests. Fix data quality first, then decide whether to retrain, recalibrate, or change business logic based on contextual analysis.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:47ML models degrade and need monitoring
- 1:27Standard ML monitoring: measuring the performance
- 1:56Standard ML monitoring is not always enough!
- 3:47How to tackle it? Early monitoring
- 4:09Data Quality: with no expectations
- 5:22Data Quality: if we have expectations
- 7:29Data Drift: compare feature distributions
- 8:18Prediction Drift: compare output distributions
- 8:56Drift detection: parametric tests
- 13:59Drift detection: large datasets
- 16:27Drift detection: non-batch models
- 17:15How to choose metrics and tests?
- 19:33Nuanced interpretations of drift
- 21:19Drift Detected, what is next?
related talks
26mProduction Data to the Model: “Are You Getting My Drift?”
34mTraining & Monitoring AI - Drift Detection
18mReliable Maintenance of Machine Learning Models
42mTesting a Data Science Model
12mModel Monitoring: Detecting and Analyzing Data Issues
36mThe ML Monitoring Flow for Models Deployed to Production
23mHow to detect silent failures in ML models
14mEarly Test Strategies in ML Model Development Phase