Handling predictive uncertainty in Machine Learning
summary
Shows concrete recipes: use scikit-learn probabilities, score models with log loss and Brier score (strictly proper scoring rules) and tune hyperparameters to get well‑calibrated predictors; derive a closed‑form threshold from a four‑entry cost matrix to make optimal binary decisions (fraud detection, medical screening, recommenders, battery dispatch). If calibration fails, apply isotonic calibration or run ThresholdClassifierCV to tune thresholds by cross‑validation. Slides and tutorial notebooks back the claims and include code and experiments that validate the approach on realistic examples.
AI-written summary · about
chapters (13)
- 0:00Predictive uncertainty in ML
- 1:10Business value and ML pipeline
- 2:59Probabilistic predictions and decisions
- 6:48Fraud detection use case
- 9:18Medical screening and recommender systems
- 12:49Battery grid optimization case study
- 16:03Optimal decision under uncertainty
- 21:30Binary classification and calibration
- 28:44Evaluating calibration with curves
- 32:35Causes of calibration issues
- 37:33Hyperparameter tuning for calibration
- 40:21Non-binary decisions and regression
- 41:04Q&A
related talks
28mGet the best from your scikit-learn classifier
40mDecision-Making Under Miscalibration
36mQuantifying uncertainty in Machine Learning predictions
31mUncertainty quantification: How much can you trust your machine learning model?
01h 10mBayesian Optimization and Machine Learning for Accelerating Experiments in the Physical Sciences
37mCreating correct and capable classifiers
01h 33mUnderstanding and diagnosing your machine-learning models
35mUncertainty Unleashed