the idea behind Automatic Relevance Determination and Bayesian Interpolation
summary
Explains the actual claim that Automatic Relevance Determination recovers sparse weights by placing independent precisions on each coefficient and selecting them by evidence, derives the math from linear models and Bayesian interpolation, and shows concrete tools: BayesianRidge and ARDRegression in Scikit-Learn, an 100×100 design matrix with 10 nonzero weights reconstruction, the closed‑form updates for alpha and lambda, and an iterative lambda pruning loop that converges and mostly recovers the true nonzero features.
AI-written summary · about
chapters (20)
- 0:00Bayesian model selection intro
- 1:29Overfitting danger in models
- 3:57Complexity versus fit balance
- 5:43Occam’s razor in Bayesian
- 7:57Evidence and model comparison
- 9:25Two levels of Bayesian
- 11:10Approximating evidence integral
- 14:05Evidence and model complexity
- 16:33Bayesian linear regression
- 18:19Bayesian Ridge Regression
- 19:51Automatic Relevance Determination
- 21:54ARD with scikit-learn
- 25:31ARD mathematical derivation
- 28:16ARD implementation in scikit
- 30:50Making predictions with ARD
- 32:17Further study resources
- 33:24Q&A: OLS vs ARD
- 34:49Q&A: ARD scalability
- 35:53Q&A: Hyperpriors and distributions
- 36:41Q&A: Nonparametric extensions
related talks
49mSo you think you know about linear regression ...
44mWinning with Simple, even Linear, Models
50mFrom Classical Statistics to Modern Machine Learning
43mRobustness in High-Dimensional Inference Tasks
44mActive Regression via Linear-Sample Sparsification
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2
45mHigh Dimensional Robust Sparse Regression
01h 18mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 1