ML: Beyond confusion matrix, model performance analysis
summary
Machine learning evaluation often fails in production because metrics and data hide failure modes: a pneumonia classifier trained on a 13k pneumonia set reaches 0.85 accuracy yet learns that asthma lowers mortality due to ICU-admission bias; an X‑ray model trained on ~40k patients scores 0.7 internally and 0.3 externally because it learns hospital markers not pathology. Use saliency (activation) maps, Facets-style dataset comparison and counterfactual checks, report class-wise confidence intervals (1,000 samples → ±~0.04) and sample thoughtfully.
AI-written summary · about
chapters (15)
- 0:00Introduction to model analysis
- 1:39Why standard metrics fail
- 5:33Fixing flawed evaluation metrics
- 8:23Business impact of model errors
- 9:07ML user experience challenges
- 11:58Data validation for models
- 15:29Fairness in data collection
- 17:17Bias detection in datasets
- 19:23Intersectional bias analysis
- 21:13Confidence intervals in testing
- 24:26ROC curve and model confidence
- 26:37Pneumonia detection failure case
- 29:52Interpretability with saliency maps
- 30:35Adversarial robustness limits
- 33:32About the speaker
related talks
40mMachine Learning Model Evaluation Metrics
43mQuality Of Machine Learning Models
15mMachine Learning: 5 Additional Metrics to Avoid High‑Performing but Useless Models
22mHow to Detect Silent Failures in ML Models
01h 33mUnderstanding and diagnosing your machine-learning models
30mHow to Properly Test ML Models & Data
27mModel Evaluation for Humans
04mBlack Box: How AI will Amplify the Best and Worst of Humanity