On continual learning with gradient descent for neural networks
summary
Analyzes continual learning for a one-hidden-layer quadratic network trained by gradient descent on an XOR cluster with Gaussian noise (task means orthogonal). Proves explicit bounds on forgetting (train and population/test-time) as functions of iterations T, samples N, tasks K and width M, using the kernel/NTK linearization plus stability-based generalization; shows forgetting can be driven to zero with choices like T ≈ D^2 and N ≈ D^2 K for large M, and verifies scalings with simulations.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 4:56Neural network theory overview
- 13:07Classification and stability analysis
- 29:05Continual learning dynamics
- 41:04Continual learning analysis
- 58:21Conclusion and outlook
related talks
01h 20mThe elusive generalization: classical bounds to double descent to grokking
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
01h 02mTutorial: Statistical Learning Theory and Neural Networks II
20mFeature Learning: From Optimization to Modern Generalization
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 04mLearning Staircases
45mFollow the Leader with Dropout Perturbations
56mHow neural networks learns simple functions?