Uncertainty Unleashed
summary
Claims that conformal prediction gives provable marginal 90% coverage on finite data and is model‑agnostic; demonstration wraps a classifier on MNIST using Python libraries (creps 0.8) and shows overall coverage ~90% but a rare class drops to 84.1% under imbalance. Calibrating per‑class raises that rare class to ~96% at the expense of average prediction set size growing (reported ~0.691 → ~1.44). Requires exchangeable calibration/test data; empty sets and larger sets are possible tradeoffs.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:47From point predictions to prediction sets
- 5:50What is conformal prediction
- 8:19Conformal prediction in workflow
- 9:25Key benefits of conformal prediction
- 10:28Understanding exchangeability
- 12:39How conformal prediction works
- 15:50Conformal prediction in 4 lines
- 16:37MNIST example
- 19:23Imbalanced data with conditional coverage
- 22:34Approximating conditional coverage
- 24:49Assumptions and limitations
- 26:13Choosing nonconformity scores
- 26:55Interpreting prediction sets
- 27:57Rethinking imbalance
- 28:52Summary
- 29:13Q&A: Set sizes and empty sets
- 31:43Q&A: Coverage interpretations
related talks
31mUncertainty quantification: How much can you trust your machine learning model?
36mQuantifying uncertainty in Machine Learning predictions
44mAccuracy Is Not Enough: Building Trustworthy AI with Conformal Prediction
30mConformal Prediction: uncertainty quantification to humanise models
37mQuantifying Uncertainty in Time Series Forecasting
30mBoosting AI Reliability- Uncertainty Quantification with MAPIE
44mHandling predictive uncertainty in Machine Learning
22mHow to Detect Silent Failures in ML Models