Diagnosing Machine Learning Models
summary
Linear OLS on 17,000 rows and ~40 inputs shows huge coefficient variance and many nonsignificant p values until one-hot encoding and redundant month/season features are fixed. Diagnostics use clustered correlation plot (absolute correlation clustered with affinity propagation via scikit learn), statsmodels condition number, pandas get_dummies with drop first, and occasional elastic net regularization. Response distribution analysis of classifier scores produces histograms that expose leakage, sparsity, overconfidence, robots and distribution shift and proves useful in production monitoring.
AI-written summary · about
chapters (17)
- 0:00Model diagnosis in ML
- 1:51Linear models nonsignificant coefficients
- 6:07Multicollinearity with correlation plots
- 10:43Fixing perfect multicollinearity
- 12:30Hierarchical multicollinearity in features
- 16:43Finalizing bicycle demand model
- 20:11Purpose-driven model diagnosis
- 20:32Traveler classification challenge
- 22:38Response distribution analysis
- 24:24Response distribution histograms
- 27:58Overconservative and sparse models
- 30:09Extreme probability predictions causes
- 32:56Monitoring model drift
- 34:44Handling incomplete dynamic data
- 36:52Limitations of response analysis
- 37:34Iterative model diagnosis
- 38:37Closing remarks Q&A
related talks
01h 33mUnderstanding and diagnosing your machine-learning models
01h 24mInpsect and try to interpret your scikit-learn machine-learning models
35mML: Beyond confusion matrix, model performance analysis
35mVisual Diagnostics for More Effective Machine Learning
57mHow To Evaluate and Improve Machine Learning Models
27mDemystifying Machine Learning
40mMachine Learning over Small Data: generating new variables with predictive power
27mPredicting Faulty Validations in Cluster Issue Detection: A ML Approach