Modern Paradigms in Generalization Boot Camp 2024
2024-08-27..2024-08-30
Berkeley, United States
who it's for
machine learning researchers, statisticians, research scientists, phd students in machine learning, applied machine learning engineers
topics
statistical learning theory and complexity; double descent in overparameterized models; degrees of freedom and SURE; diffusion model hallucination detection; domain adaptation transfer exponent; contrastive pretraining and spectral clustering; robust estimation under Huber contamination; statistical limits of causal inference; density estimation sample complexity; score-based loss generalization; multi-objective machine learning algorithms; feature formation and recursive feature machines
| title | author | length |
|---|---|---|
| Prediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 1 | Ryan Tibshirani | 01h 18m |
Reviews classical fixed‑X decision theory then shows it often fails for random‑X interpolation regimes and how to extend concepts. Derives the optimism = 2 ·… | ||
| Overview of Statistical Learning Theory Part 2 | Nati Srebro | 01h 03m |
Surveys classical tools and then ties them to optimization: uniform convergence via fat-shattering dimension and Rademacher complexity gives sample bounds… | ||
| Understanding (a bit about) hallucinations in Generative AI | Zico Kolter | 01h 15m |
Diffusion models trained on a simple shapes dataset and on hands produce 1–2% clearly impossible samples (two shapes in one region, nonfive fingers). Analysis… | ||
| Measures of relatedness in domain adaptation and related problems. | Samory Kpotufe | 01h 15m |
Proposes a concrete quantity called the transfer exponent and a broader modulus of continuity that map source excess risk to target excess risk, proves minimax… | ||
| The elusive generalization: classical bounds to double descent to grokking | Misha Belkin | 01h 20m |
Large neural models and a simple iterative kernel method produce the same surprising pattern: training loss is essentially zero while validation/test accuracy… | ||
| Reconsidering Overfitting in the Age of Overparameterized Models | Fanny Yang | 01h 18m |
Empirical and theoretical claims: overparameterized linear models show double descent because variance decays as dimension grows while bias rises; for… | ||
| A Theory of Multi-objective Machine Learning | Nika Haghtalab | 01h 11m |
Introduces a concrete formalism: learn one model good for k distributions (per-distribution losses and calibration), cast as a min‑max game solved by no-regret… | ||
| Learning probability distributions; what can, what can't be done | Shai Ben-David | 01h 12m |
Shows that density estimation is impossible without prior knowledge: samples of size independent of domain size cannot distinguish distributions that are 0.5… | ||
| Understanding Generalization from Pre-training Loss to Downstream Tasks | Tengyu Ma | 01h 18m |
Contrastive pretraining provably produces embeddings where contrastive loss → spectral clustering and Euclidean distance ≈ manifold distance on the population… | ||
| Statistical Limits of Causal Inference | Sivaraman Balakrishnan | 01h 11m |
Covers three concrete vignettes: limits for personalized causal effects (when tau is smooth but mu0/mu1 are not), rates without any smoothness where one-step… | ||
| An overview of classical robust statistics and generalizations to the future | Po-Ling Loh | 01h 15m |
Huber theory is presented with precise claims: Huber loss is minimax-optimal for asymptotic variance under Huber ε‑contamination (γ is tuned to ε), the… | ||
| Prediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2 | Ryan Tibshirani | 01h 02m |
Shows that classical fixed-X results break down once models approach interpolation and X is random, and gives precise tools: an exact covariance formula for… | ||
| Generalization in natural language processing | Alane Suhr | 01h 16m |
Survey of what NLP actually generalizes to: charts the shift from lexicon-plus-grammar pipelines to large self‑supervised models, shows transformers trained on… | ||
| Overview of Statistical Learning Theory Part 1 | Nati Srebro | 01h 17m |
Survey of classical statistical learning theory that pins down learnability: VC dimension and Sauer's Lemma give tight sample complexity and explain the sharp… | ||
| Towards Understanding Generalization Properties of Score-Based Losses | Andrej Risteski | 01h 17m |
Connects the statistical loss of score-based fitting to the sampling cost of the inference chain: parameter error of score matching is bounded by MLE error… | ||
| Modern paradigms of generalization, the heliocentric model of Aristarchus,... | Matus Telgarsky | 01h 10m |
Four concrete threads: formalize why next-token prediction yields broad downstream performance and why GPT vs BERT differ; explain LLaMA 3’s empirical trick of… | ||
| Robust Optimization and Generalization | John Duchi | 01h 17m |
A production LP with tiny input shifts (about 0.5%–2%) shows losses of 15%–20% when reusing the same plan; a robust LP that enforces rowwise perturbations… | ||
| nothing matches | ||