Get the best from your scikit-learn classifier
summary
Scikit-learn adds a meta-estimator that fits classifiers for calibrated probabilities then tunes the decision threshold to directly maximize the chosen metric (including business cost matrices), rather than relying on the default 0.5 cutoff or blind resampling. The workflow emphasizes using optimize log loss (proper scoring rule) during hyperparameter search, recalibrating if needed, and then selecting the threshold that yields the best hard decisions; experiments show tuned thresholds plus proper scoring match or beat naive resampling. See PR: https://github.com/scikit-learn/scikit-learn/pull/26120.
AI-written summary · about
chapters (7)
- 0:00Imbalanced data challenges
- 3:13Why resampling is not a magical solution
- 7:13Calibration after resampling
- 12:35Hyperparameter tuning
- 16:28Metadata routing for metrics
- 20:39Key takeaways
- 22:02Q&A
related talks
44mHandling predictive uncertainty in Machine Learning
48mOptimize the Right Thing
31mAn update on the latest scikit-learn features
42mCreating correct and capable classifiers
40mDecision-Making Under Miscalibration
44mResampling techniques and other strategies
01h 04mPredicting and optimizing the behavior of large ML models
37mCreating correct and capable classifiers