Spectral Algorithms for Learning Latent Variable Models
summary
Spectral methods use explicit moment equations and tensor algebra to recover latent mixtures: three-word documents suffice for identifiable topic vectors provided topics are linearly independent and K ≤ D, whitening plus a tensor power method with a few restarts recovers components with provable perturbation bounds. For spherical Gaussians the variance is the smallest eigenvalue and subtracting it reduces to the same scheme. Zero-mean mixtures need higher moments; current provable algorithms require K up to about D^2 (smooth-analysis) though higher moments expect K ≈ D.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:34Two canonical examples
- 2:36What are the limits of learning?
- 3:39A Different Approach
- 5:14Related Work
- 6:18Mixture Models
- 11:45The Method of Moments
- 13:28vector notation and multinomials!
- 14:30With the first moment?
- 16:39With the second moment?
- 23:21With three words per document?
- 25:02Tensors and Linear Algebra
- 25:35Tensor eigenvectors
- 26:48Estimation
- 28:29Algorithm: Tensor Power Iteration
- 34:13Mixtures of spherical Gaussians Theorem
related talks
53mTensor Decompositions for Learning Latent Variable Models II
56mTensor Decompositions for Estimating Latent Variable Models
52mLearning Mixture of Gaussians using Moment Tensor
56mTensor Decompositions for Learning Latent Variable Models I
54mLearning Overcomplete Latent Variable Models through Tensor Power Method
01h 31mUnsupervised Learning
01h 03mRecent Progress in High-Dimensional Learning
44mMixture Models, Robustness, and Sum of Squares Proofs