Tensor Decompositions for Learning Latent Variable Models I
summary
Claims that a mixture of multinomial topic models with one topic per document is identifiable from third-order moments when topic word-vectors are linearly independent and weights are positive, while pairwise (two-word) statistics are insufficient. Recovery uses Jenrich's algorithm — collapse the empirical 3-way tensor with two random vectors, form a matrix, and read off eigenvectors — and scales when K ≤ D. Applied to NYT (vocab ~100k, K=50) and matches Gibbs-LDA performance; discusses sample, conditioning and robustness limits.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:45Example #1: summarizing a corpus of documents
- 1:44Topic model (eg, latent Dirichlet allocation)
- 4:12Labels / annotations
- 6:51Example #2: subpopulations in data
- 9:44Maximum likelihood estimation
- 11:14MLE for Gaussian mixture models
- 13:19Parameter learning objective
- 19:31Ways around the barriers
- 21:10Method-of-moments at a glance
- 24:05Unresolved issues
- 25:25Outline
- 26:18Other models amenable to moment tensor decomposition
- 33:32Identifiability from documents of length three
- 34:32Tensors of order p
- 37:30Jennrich's algorithm (simplified)
- 44:48Application to topic model parameters
- 46:06Recap
- 49:46Illustrative empirical results
- 53:37Learning algorithms
related talks
56mTensor Decompositions for Estimating Latent Variable Models
52mTensor Methods for Learning Latent Variable Models: Theory and Practice
01h 03mRecent Progress in High-Dimensional Learning
49mSpectral Algorithms for Learning Latent Variable Models
58mTensor Decomposition, Sparse Representations and Applications
01h 11mTensor Decompositions: Uniqueness and Smoothed Analysis
54mLearning Overcomplete Latent Variable Models through Tensor Power Method
01h 00mSmoothed Analysis of Tensor Decompositions