How to Detect Silent Failures in ML Models
summary
Open-source Python library NannyML implements a concrete method: it calibrates model probabilities, applies a decision threshold, builds per-row partial confusion matrices (e.g., 0.9 true positive, 0.1 false positive) and sums them to produce confidence-based performance estimates when labels are missing. Multivariate drift detection uses dimensionality reduction (PCA) plus reconstruction error to catch changes in feature relationships that univariate tests miss. Examples: labels often arrive after 3–5 years or only ~1% are spot-checked; estimation matches realized performance on a California Housing binary conversion.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:27Setting the Stage: Loan Default Prediction
- 6:02Concept drift
- 6:51Performance matters.
- 15:07Performance Estimation - California Housing dataset
- 15:53Data Drift Detection
- 18:28Data reconstruction
- 20:10PCA Reconstruction error - examples
related talks
22mHow to Detect Silent Failures in ML Models
23mHow to detect silent failures in ML models
36mThe ML Monitoring Flow for Models Deployed to Production
12mOn the Use of ML for Blackbox System Performance Prediction
18mReliable Maintenance of Machine Learning Models
35mA simple way to validate and monitor the performance of your ML Applications.
35mML: Beyond confusion matrix, model performance analysis
11mLearning from your (model's) mistakes