Are Generative Models The New Sparsity?

on YouTube ↗29m

" summary

Generative priors sampled from random-weight neural nets reduce the algorithmic–statistical gap seen with sparsity in toy high-dimensional inference: rigorous MMSE formulas via mutual information show that AMP matches Bayes optimal in the spiked matrix model, and AMP-derived LAMP spectral estimator outperforms PCA on real-image covariances. For phase retrieval and compressed sensing AMP still finds hard phases, but those hard phases shrink with depth and compression and disappear asymptotically in the models studied.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 1:51Inverse problems
3. 3:08Examples
4. 3:33Sparsity
5. 4:30Learning a basis
6. 6:22Matrix factorisation
7. 7:32Sparse case
8. 8:55Generative networks
9. 9:38Statistical guarantee
10. 10:48Role of depth
11. 11:25Compositional Principle
12. 12:23AMP for one-layer prior
13. 15:04Spectral method
14. 17:16LAMP on Fashion MNIST
15. 17:51Other problems
16. 18:29Generalised linear estimation

" related talks

• 46mSolvable Model of Unsupervised Feature LearningLenka Zdeborova · Random Instances and Phase Transitions (Simons Institute, 2016)
• 19mGeneralSparse: Bridging the Gap in SpMM for Pruned Large Language Model Inference..Yaoyu Wang, Xiao Guo, Junmin Xiao, De Chen, Guangming Tan · USENIX ATC '25
• 52mDeep Generative Models And Unsupervised Methods For Inverse ProblemsAlex Dimakis · Algorithmic Advances for Statistical Inference with Combinatorial Structure
• 58mUniversality of Approximate Message Passing on Semi-random MatricesSubhabrata Sen · Deep Learning Theory Workshop and Summer School 2022
• 40mSparse models are fast models: Improving DNN inference performance by over 10XLawrence Spracklen · Data Con LA 2021
• 01h 04mStatistical Physics and Computation in High DimensionFlorent Krzakala, Lenka Zdeborova · Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020)
• 31mStatistical Estimation on GraphsAndrea Montanari · Coding: From Practice to Theory
• 01h 06mTutorial: Methods from Statistical Physics IIAhmed El Alaoui · Deep Learning Theory Workshop and Summer School 2022