Subtractive Mixture Models: Representation and Learning

on YouTube ↗52m

" summary

Introduces subtractive mixtures by allowing negative mixture weights, then enforces non-negativity by squaring the circuit, producing squared nonmonotonic circuits that expand K components to O(K^2) via cross-terms; partition functions remain tractable when circuits are structured decomposable so renormalization is feasible and GPU-friendly via layerwise tensor/Kronecker operations. Proves an exponential separation (lower bound) versus monotonic PCs, shows empirical log‑likelihood gains on real datasets (including GPT-2 token distributions with large vocabularies), implements sampling by autoregressive inverse‑transform, and flags latent‑variable semantics and structure learning as open issues.

AI-written summary · about

" chapters (14)

1. 0:00Subtractive mixture models intro
2. 3:18Mixture models overview
3. 10:10Subtractive models with negatives
4. 14:49Squaring circuits for non-negativity
5. 19:30Deep structured circuits squaring
6. 23:09Tensorized circuits for GPU
7. 30:11Connections to kernels and networks
8. 34:11Inference with nonmonotonic circuits
9. 36:45Expressivity gains from squaring
10. 38:30Lower bounds for subtractive models
11. 40:41Empirical evaluation
12. 46:21Open problems
13. 48:08Summary
14. 49:16Q&A

" related talks

• 40mIdentifying Mixture ModelsLeonard J. Schulman · Algorithmic Aspects of Causal Inference 2022
• 01h 05mLogic and Probabilistic Circuits 1Guy Van den Broeck · Logic and Algorithms in Database Theory and AI Boot Camp
• 37mTractable Probabilistic CircuitsGuy Van den Broeck · Algorithmic Aspects of Causal Inference 2022
• 44mMixture Models, Robustness, and Sum of Squares ProofsJerry Li · Robust and High-Dimensional Statistics (Simons Institute workshop, 2018)
• 53mTensor Decompositions for Learning Latent Variable Models IIDaniel Hsu · Foundations of Machine Learning Boot Camp 2017
• 32mNew Approaches To Learning Nonparametric (Latent) Causal Graphical ModelsBryon Aragam · Algorithmic Aspects of Causal Inference 2022
• 01h 03mLogic and Probabilistic Circuits 2YooJung Choi · Logic and Algorithms in Database Theory and AI Boot Camp
• 29mCompositional sparsity and learnabilityTomaso Poggio · Smale@95: A Conference in Honor of Steve Smale