Feature Importance and Ensemble Methods : a new perspective
summary
Empirical claim: the default tree split-count importance (scikit-learn style) often ranks useless fields like ID or injected noise highly — in one XGBoost run pure uniform noise becomes random noise ranks third while model AUC stays unchanged. Tools and fixes: experiments with Random Forests and XGBoost, a gain×cover-weighted importance and a per-instance pointwise feature scores algorithm. Results: weighted global scores push noise down but not entirely; pointwise scores expose drivers of false positives in 4/10 cases. Limitations: correlated features, no decision thresholds, and no causal guarantees.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 4:00Feature importance in ML
- 7:11Ensemble methods explained
- 10:20Limitations of current metrics
- 13:52Gradient boosting mechanics
- 16:43Robust feature importance metric
- 19:35Testing new metric on data
- 21:24Point-to-point feature importance
- 24:34Linking scores to predictions
- 26:01Point-level importance for business
- 27:48Drawbacks of point-level importance
- 29:54Correlated variables impact
- 31:24Conclusion
- 33:08Q&A
related talks
10mUsing Dominance Analysis for accurate feature importance
01h 05mEvaluating Model Explainability Methods
29mOpeningTheBlackBox
30mMachine Learning Techniques for Class Imbalances & Adversaries
44mUnderstanding Random Forests
38mWhy giving your algorithm ALL THE FEATURES does not always work
44mEnsemble Techniques for High Performance Machine Learning
48mInherent Trade-offs with the Local Explanations Paradigm