An Instability in Variational Methods for Learning Topic Models
summary
Variational mean-field inference on low-rank generative models produces a spurious informative fixed point between the information-theoretic spectral threshold and a lower algorithmic instability: in the Z2 sync toy the uninformative posterior becomes unstable at lambda = 1/2 even though no estimator can correlate with the truth until lambda = 1. This conclusion follows from Hessian/spectrum calculations, state-evolution and large-scale simulations; replacing naive mean field with the TAP/AMP correction restores stability up to the true threshold.
AI-written summary · about
chapters (18)
- 0:00Introduction to topic models
- 3:25Variational inference
- 7:04Optimization problem formulation
- 10:23Z2 synchronization example
- 15:37Analyzing variational inference
- 22:49Uninformative point instability
- 25:55Topic model applicability
- 28:36Threshold analysis
- 30:18Calculation for topic models
- 37:12Algorithm instability results
- 43:38Numerical simulations
- 45:46Correlation analysis
- 46:47Bender cumulant results
- 49:06Summary of results
- 49:21Improved algorithm results
- 50:27Stability of improved methods
- 51:59Final conclusions
- 53:34Questions and answers
related talks
45mTAP Free Energy, Spin Glasses, and Variational Inference
49mSpectral Algorithms for Learning Latent Variable Models
46mSolvable Model of Unsupervised Feature Learning
29mAre Generative Models The New Sparsity?
51mRandom Projections for Probabilistic Inference
35mThresholds of Descending Algorithms in Inference Problems
01h 06mTutorial: Methods from Statistical Physics II
57mTutorial on Deep Learning III