Learning One-hidden-layer Neural Networks with Landscape Design
summary
Designs a new population objective g that uses Hermite-polynomial coefficients to match Hermite coefficients between training activation gamma and data activation sigma, so all unwanted terms vanish and g reduces to only second- and fourth-order tensors. Proves for Gaussian inputs, orthogonal B and positive A that all local minima of g are global and gradient methods recover parameters; empirically it converges to zero parameter error where plain L2 stalls but needs much larger batches and more samples and breaks if inputs are non-Gaussian.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:39Interfaces Between Users and Optimizers?
- 2:13Optimization in Machine Learning: New Interfaces?
- 5:29Possible Paradigm for Optimization Theory in ML?
- 7:05This Talk: New Objective for Learning One-hidden-layer Neural Networks
- 9:56The Straightforward Objective Fails
- 13:58An Analytic Formula
- 21:23Provable Non-convex Optimization Algorithms?
- 28:03Conclusion
related talks
56mHow neural networks learns simple functions?
01h 02mTutorial: Statistical Learning Theory and Neural Networks II
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 30mAnalytic Methods for Supervised Learning IV
01h 06mAnalytic Methods for Supervised Learning III
32mRobust Learning of a Single Neuron via Sharpness
32mLearning Ill-Conditioned Gaussian Graphical Models
19mOn the Cryptographic Hardness of Learning One-Hidden Layer Neural Networks