When is Scale Enough?
summary
Analytic linear/kernel models produce a power-law kernel spectrum that predicts test-error scaling and yields four regimes (data-rich vs over-/under-parameterized, variance- vs resolution-limited) with exponents set by kernel decay; Big‑Bench shows many tasks improve smoothly with model size while others show abrupt turn‑ons or depend on prompt/scoring choices; large unsupervised pretraining plus scaled data measurably pretraining reduces forgetting across fine‑tuning sequences; math-heavy further pretraining plus chain-of-thought + majority voting boosts quantitative reasoning but still gives false positives, memorization, and brittle generalization.
AI-written summary · about
chapters (13)
- 0:00Start
- 3:20Review Scaling in Linear Models
- 3:55Teacher Student Setup
- 26:06Why Do We Need a New Benchmark
- 28:29Examples of the Types of Tasks
- 30:55Can We Predict Performance on Particular Tasks
- 31:24Cause and Effect Task
- 42:55Conclusion
- 44:33The Typical Catastrophic Forgetting Setup
- 46:38The Forgetting Frontier
- 56:48Majority Voting
- 59:23Asymptote
- 1:01:36Failure Modes
related talks
49mPanel Discussion
36mWhen the Only Way to Scale up, is to Scale Down
33mThe Power and Limitations of Kernel Learning
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
26mThe scaling laws of AI Why neural networks continue to grow
46mKernel and Deep Regimes in Overparameterized Learning
47mScale Calibration for High-Dimensional Robust Regression
42mToward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain