Learning from your (model's) mistakes
summary
Claims that a secondary binary model—Model Performance Predictor—learns to flag failure cases by training a decision tree on test features labeled success/failure and thereby clusters errors into meaningful nodes. Regression failures use a knee-point tolerance; classification failures split by false positive/negative per class. An open source error analyzer plus visualizer is built to train the MPP on pipeline models, highlight large pure failure nodes and correlated features, and predict likely failures on unlabeled data.
AI-written summary · about
chapters (9)
- 0:00Model mistakes analysis
- 0:46Model debugging essentials
- 1:28Model errors clusters
- 2:30Performance predictors
- 4:56Insights from failure nodes
- 7:03Interpreting features
- 8:03Error analysis tool
- 9:27Practical applications of performance predictors
- 10:30Q&A
related talks
22mHow to Detect Silent Failures in ML Models
19mModel Performance Monitoring - A Practitioner's Perspective- 22mHow to Detect Silent Failures in ML Models
35mWhat Failure Taught Me About Building High-Stakes Models
23mHow to detect silent failures in ML models
32mReliable Machine Learning
35mML: Beyond confusion matrix, model performance analysis
37mCreating correct and capable classifiers