Deep Learning Theory Workshop and Summer School 2022
2022-08-01..2022-08-04
Berkeley, United States
who it's for
deep learning theorists, machine learning researchers, graduate students in machine learning, statistical physicists, computational biologists, algorithmic statisticians, optimization researchers
topics
VC dimension for ReLU nets; Rademacher complexity bounds; implicit bias of gradient methods; neural tangent kernel regime; feature learning and linear probing; benign and tempered overfitting; distribution shift and domain adaptation; approximate message passing (AMP) theory; replica and cavity methods; convolutional kernels on images; high-dimensional causal inference; deep learning for protein design
| title | author | length |
|---|---|---|
| Tutorial: Statistical Learning Theory and Neural Networks I | Spencer Frei | 59m |
Review of uniform laws of large numbers and concrete complexity tools: VC dimension (upper bound ≈ O(P·L) for ReLU nets) and Rademacher complexity, with a… | ||
| Tutorial: Statistical Learning Theory and Neural Networks II | Spencer Frei | 01h 02m |
Shows that gradient-based training either satisfies a Polyak–Lojasiewicz (PL) inequality (so stationary points imply small empirical exponential loss) under… | ||
| Understanding the Robustness of Deep Learning | Aditi Raghunathan | 01h 00m |
Pre-trained models help robustness, but the adaptation rule matters: with limited labeled adaptation data and overparameterized features, updating all weights… | ||
| Feature Selection with Gradient Descent on Two-layer Networks in Low-rotation Regimes | Matus Telgarsky | 58m |
Gradient flow and SGD on two‑layer ReLU nets show concrete selection and generalization claims in low‑rotation regimes: near initialization they exit the NTK… | ||
| Tutorial: Implicit Bias I | Nati Srebro | 01h 21m |
Claims that classic statistical learning tools largely apply to deep learning while implicit bias and benign overfitting need refined views; uses Rademacher… | ||
| The Devil is in the Tails and Other Stories of Interpolation | Niladri Chatterji | 55m |
Shows that reweighting fails for exponentially-tailed losses but recovers when using a polynomially-tailed loss, with theory and CIFAR/CelebA runs verifying… | ||
| Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting | Neil Mallinar, Preetum Nakkiran | 58m |
Defines a three-way taxonomy — benign overfitting, tempered overfitting, catastrophic — and claims many interpolating deep nets sit in the middle: nonzero but… | ||
| Tutorial: Implicit Bias II | Nati Srebro | 01h 37m |
Experiments fit MNIST nets with 8,16,64,512,1000+ hidden units and show zero training error on random labels (Zhang et al.), yet test error keeps falling… | ||
| Tutorial: Methods from Statistical Physics I | Ahmed El Alaoui | 59m |
Introduces the Gaussian additive model with explicit posterior as a Gibbs measure and defines partition function Z and free energy F = E log Z; shows F′ and F″… | ||
| Tutorial: Methods from Statistical Physics II | Ahmed El Alaoui | 01h 06m |
Dense rank‑one Gaussian model (spiked Wigner): observations Y = (λ/n) x0 x0^T + W, x0 i.i.d. prior. Bayes free energy obeys a replica symmetric formula… | ||
| Distribution Shift as Underspecification, and What We Might Do About It | Chelsea Finn | 01h 01m |
Proposes DivDis which trains multiple hypotheses by minimizing training loss while maximizing disagreement on unlabeled target examples then disambiguates with… | ||
| A New Perspective on High-Dimensional Causal Inference | Pragya Sur | 57m |
Paper proves a new central limit theorem for a three‑split cross‑fit AIPW estimator in a proportional regime (p/n → constant) without sparsity, explicitly… | ||
| Tutorial: Methods from Statistical Physics III | Ahmed El Alaoui | 01h 08m |
Derives the AMP memory term by converting pairwise message passing on the complete graph into a bulk update, tracking backtracking loops and using A_ij ~… | ||
| A Theoretical Framework of Convolutional Kernels on Image Datasets | Song Mei | 01h 00m |
Characterizes the RKHS of single‑layer convolution + pooling + downsampling and analyzes kernel ridge regression and random‑feature limits under isotropic data… | ||
| Universality of Approximate Message Passing on Semi-random Matrices | Subhabrata Sen | 58m |
Claim: AMP iterates on semi‑random matrices M = S Ψ S (diagonal ±1 signing with deterministic Ψ that is delocalized and ΨΨᵀ ≈ σ²I entrywise) produce the same… | ||
| When is Scale Enough? | Ethan Dyer | 01h 12m |
Analytic linear/kernel models produce a power-law kernel spectrum that predicts test-error scaling and yields four regimes (data-rich vs… | ||
| Deep Learning in Structural Biology and Protein Design: How, Where, and Why | Chloe Hsu | 01h 13m |
Explains how AlphaFold2 combines MSAs and an EvoFormer with recycling iterative refinement to predict all-atom coordinates and outputs a confidence score well… | ||
| nothing matches | ||