Learning by Local Entropy Maximization
summary
Claim: random binary perceptrons hide rare local entropy maxima—exponentially dense clusters of solutions that standard replica analyses miss. Tools: large-deviation replica calculations, Franz–Parisi style potentials, belief propagation and a replicated Monte Carlo /replicated-gradient construction. Surprise: dense clusters persist up to a critical load alpha ≈ 0.77 (numerics show disappearance near 0.756–0.77), experiments on sizes up to thousands (e.g. N≈6400) and on MNIST show simple replicated algorithms find solutions efficiently and generalize, while plain simulated annealing gets trapped.
AI-written summary · about
chapters (5)
- 0:00Start
- 3:05"Old"results from statistical physics
- 5:14Notation
- 5:44Learning as a CSP problem
- 39:02Multiple layers
related talks
01h 08mTutorial: Methods from Statistical Physics III
01h 04mStatistical Physics and Computation in High Dimension
54mA Curious Case Of Symmetric Binary Perceptron Model: Algorithms And Barriers
01h 11mStatistical Physics and Computation in High Dimension
01h 04m"More is Different." Also in Computing?
01h 03mStatistical Physics and Computation in High Dimension
01h 07mTheory of Computation II
15mFundamental Limits Of Learning In Data-Driven Problems