Recent Developments in Over-parametrized Neural Networks, Part I
summary
Shows that widening a network often fixes optimization: a realizable two‑layer teacher with 50 neurons resists SGD when fitted with 50 units but reaches near‑zero population loss (~0.01) when fitted with 100 units, with a sharp phase transition around ~55–70. Analysis splits into geometry/topology (no spurious minima if a layer is super‑wide), local linearization/NTK regimes (Polyak gradient domination gives linear convergence near init), and escape‑saddle dynamics (perturbed SGD finds second‑order stationary points). Practical fixes like ResNet skip connections improve trainability.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 0:27Outline
- 1:59Goals
- 2:32Whats Hard
- 3:32Whats Easy
- 4:32Non convex optimization
- 7:37Notation
- 8:51Overparametrization
- 11:20Experiments
- 16:49Rule of Thumb
- 17:23Theoretical Problem
- 18:36Optimal Optimization
- 19:43Taxonomy of Results
- 20:53Skip Connections
- 21:51Expressivity
- 22:30Geometric Results
- 22:50Geometry Results
- 25:58Onepoint convexity
- 32:06Taylors theorem
- 32:57Royer algorithm
- 38:31Randomness
- 40:03Theorem
- 42:17Secondorder Stationary Points
- 43:41Neural Networks
- 48:58Optimality Conditions
- 50:28Global Optimal
- 51:43Other Losses
- 52:30NonDegenerate Critical Points
- 53:12Three Strategies
- 56:28Second Order Descent
- 57:24Stationary Points
- 58:57Learning Networks
related talks
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
28mTowards the Universal Law of Robustness for Deep Neural Networks
46mKernel and Deep Regimes in Overparameterized Learning
19mTraining (Overparametrized) Neural Networks in Near-Linear Time
33mProvably Efficient Algorithms for Training Deep Networks
56mTutorial on Deep Learning II