Recent Developments in Over-parametrized Neural Networks, Part II

on YouTube ↗01h 14m

" summary

Survey of recent theory connecting wide nets to neural tangent kernel and random features frameworks, with concrete claims: kernel limit arises as width → ∞, Hessian terms shrink like Hessian ~ 1/√M, and linearized (NTK) dynamics match network training for square loss for all time but for logistic only until loss ≈ 1/M. Proof tools: RKHS approximations, Gaussian-process initialization recursions, and perturbation bounds. Limitations emphasized: NTK needs huge feature counts (often N^2 or worse), fails on simple single-ReLU targets unless initialization/signs are special, and requires learning rate scaling ∝ 1/M to stay in kernel regime.

AI-written summary · about

" chapters (30)

1. 0:00Schedule and topic update
2. 0:20Moving to algorithmic results
3. 1:02Limitations of current models
4. 2:16Review of random features
5. 4:12Expressivity of random features
6. 6:23Proof strategy for learnability
7. 7:20Linear representation
8. 9:06Norm constraints for approximation
9. 11:37SGD optimization analysis
10. 13:18Optimizing both neural layers
11. 15:26Activation function considerations
12. 18:57Application to deeper layers
13. 21:32Introduction to NTK
14. 24:28Neural tangent kernel definition
15. 27:16NTK parameterization families
16. 30:37Kernel closed forms
17. 32:03Kernel computation for deep nets
18. 35:46Training dynamics linearization
19. 38:05Hessian vanishing property
20. 41:11Kernel and network comparison
21. 44:10Complexity of NTK training
22. 45:37Learnable functions with NTK
23. 47:40Teacher network learning
24. 50:26Polynomial learning comparison
25. 53:35Loss function impact
26. 56:08Learning rate scaling
27. 59:12NTK limitations and gaps
28. 1:01:20Inductive bias alignment
29. 1:04:54Empirical evidence on NTK
30. 1:07:37Future research directions

" related talks

• 50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond KernelsYuanzhi Li · Frontiers of Deep Learning 2019
• 46mOn the Foundations of Deep Learning: SGD, Overparametrization, and GeneralizationJason Lee · Frontiers of Deep Learning 2019
• 19mThe Spectrum of Nonlinear Random Matrices for Ultra-Wide Neural NetworksYizhe Zhu · Deep Learning Theory Symposium 2021
• 49mComputation in Very Wide Neural NetworksYasaman Bahri · Frontiers of Deep Learning 2019
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 01h 02mTutorial: Statistical Learning Theory and Neural Networks IISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 28mTowards the Universal Law of Robustness for Deep Neural NetworksLuka Nenadović · DSC Europe 2022
• 17mUniversality of Neural NetworksDan Mikulincer · Deep Learning Theory Symposium 2021