Feature Learning: From Optimization to Modern Generalization
summary
Shows that a two-layer neural network trained with online minibatch SGD and logistic loss on Boolean hypercube data learns an XOR-style multi-index target, with neurons aligning to the true 1-D projections and convergence to a global optimum. Proof uses optimization analysis of SGD; sample complexity falls to O(d log d) samples versus the naive Θ(d^2) needed without feature learning. Empirical/constructive result is theoretical and proven for that setting.
AI-written summary · about
chapters (11)
- 0:00Welcome
- 2:28Feature learning intro
- 3:54Optimization for feature learning
- 5:19Multi-index models
- 7:24SGD learning theorem
- 9:30Sample complexity in SGD
- 10:53Future research in feature learning
- 11:56Features as modules
- 14:04Learning function compositions
- 17:15Gradient descent & generalization
- 19:24Q&A
related talks
01h 20mThe elusive generalization: classical bounds to double descent to grokking
33mProvably Efficient Algorithms for Training Deep Networks
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
32mRobust Learning of a Single Neuron via Sharpness
56mHow neural networks learns simple functions?
47mIs Minimax Optimization the Right Framework to Understand Learning in Minimax Games?
01h 04mRandomized Embeddings and Neural Networks
01h 00mOn continual learning with gradient descent for neural networks