Decision-Making Under Miscalibration
summary
Formalizes a distribution-free decision rule that picks the threshold minimizing worst-case regret over all predictors with at most anticipated miscalibration α, gives closed-form solutions under expected and maximum calibration error, and shows the safe threshold is often more conservative than the classic therapeutic threshold when that threshold lies near 0 or 1. Validates on real data including small-data (400–1,600) experiments and subgroup shift (feature-omission) and reports consistent utility gains from the regret-minimizing threshold.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:03Background
- 6:12Miscalibration
- 9:52Why Im excited
- 12:03Definitions
- 15:50Net Benefit
- 21:31Recap
- 25:29Takeaways
- 28:44Solution
- 30:07Experimental Validation
- 34:06Summary
related talks
44mHandling predictive uncertainty in Machine Learning
09mTowards Credible and Effective Data-Driven Decision-Making
29mHigh Dimensional Calibration for Rational Decision Making
01h 21mMulticalibration: Towards Fair Decision Making
17mCodifying Data Science Intuition
12mCausal Inference And Data-Driven Decision-Making
01h 11mA Theory of Multi-objective Machine Learning
48mOnline Adversarial Multicalibration And (Multi)Calibeating