You Should Probably Be Doing Nested Cross-Validation
summary
Claim: nested cross-validation gives much less biased, lower-variance estimates of final model performance on small datasets than a single train/validation/test split. Demonstrates scikit-learn code using GridSearchCV inside an outer cross_val_score loop, shows typical settings like 5–10 inner folds and 4–10 outer folds, and measures resulting spread (several percent) across outer-test folds. Notes cost multiplies by number of outer folds but is embarrassingly parallel, and suggests ensembling models from outer folds or using nested CV for medical/small-data cases.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:08Why standard CV fails
- 3:13Validation set bias
- 6:03How k-fold CV helps
- 8:30Single test set limits
- 11:42Introducing nested cross-validation
- 13:08Computational cost
- 14:54Practical applications
- 17:19When to use nested CV
- 18:22Summary
- 18:55Q&A
related talks
01h 33mUnderstanding and diagnosing your machine-learning models
28mCommon pitfalls leading to wrongly estimated model performance
44mEnsemble Techniques for High Performance Machine Learning
57mHow To Evaluate and Improve Machine Learning Models
35mScikit-Learn can do THAT?!
38mUsing Reproducible Experiments To Create Better Models
26mAsympirical Analysis: Theory Informs Practice
39mImplementing distributed grid search for deep learning using scikit learn and joblib