Deep Learning Theory Symposium 2021
2021-12-03
Berkeley, United States
who it's for
machine learning theorists, graduate students in machine learning, statisticians working on learning theory, optimization researchers, theoretical computer scientists, applied mathematicians
topics
mean-field PDEs for neural nets; neural tangent kernel limitations; nonparametric SGD convergence rates; spectral-norm matrix concentration; graphon neural network transferability; dynamical mean field theory; universality in ultra-wide networks; cryptographic hardness of one-hidden-layer learning; adaptive wavelet distillation from ResNets; uniform convergence and benign overfitting; representation costs of linear networks
| title | author | length |
|---|---|---|
| Learning Staircases | Emmanuel Abbe, Enric Boix, Theodor Misiakiewicz | 01h 04m |
Claim: certain nested “staircase” Boolean functions with growing degree are not kernel learnable (NTK/random features require ~D^K samples/features) but are… | ||
| Adaptive Wavelet Distillation from DNNs and Dictionary Learning | Bin Yu | 01h 01m |
Pretrained ResNet outputs are distilled into an adaptive wavelet transform with a three-term loss (reconstruction + wavelet constraints + an L1 gradient… | ||
| Graphon Neural Networks and the Transferability of Graph Neural Networks | Luiz Chamon | 18m |
Constructs a graphon neural network as the limit of GNNs sampled from the same graphon and proves output difference bounds that split into… | ||
| Self-Training Converts Weak Learners to Strong Learners in Mixture Models | Spencer Frei | 17m |
Shows that self-training with pseudo-labelers turns a classifier that has constant error below a fixed C_error constant into one within ε of the Bayes optimal… | ||
| Non-Parametric Convergence Rates for Plain Vanilla Stochastic Gradient Descent | Raphaël Berthier | 13m |
Proves that plain vanilla SGD—single-pass, unregularized, fixed step size—achieves nonparametric excess-risk bounds in infinite-dimensional linear/kernel… | ||
| Sharp Matrix Concentration | March Boedihardjo | 27m |
Claims removal of the usual log^2 d loss for spectral-norm concentration of Gaussian random matrices by trading it for a concrete second-order covariance term… | ||
| Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting | Frederic Koehler | 01h 04m |
Proves a uniform‑convergence bound for interpolators in Gaussian linear regression: test loss ≤ (Gaussian‑width(K))^2 / n up to small additive/multiplicative… | ||
| Exact Asymptotics and Universality for Gradient Flows and Empirical Risk Minimizers | Andrea Montanari | 54m |
Paper with Michael Celentano and Chen Chen proves that in proportional high‑dim limits one coordinate trajectory of high‑dimensional gradient flow has a… | ||
| The Spectrum of Nonlinear Random Matrices for Ultra-Wide Neural Networks | Yizhe Zhu | 19m |
Model is a two-layer fully connected network at initialization with iid Gaussian weights and centered, normalized activation; analysis targets the ultra-wide… | ||
| Universality of Neural Networks | Dan Mikulincer | 17m |
Result gives a non-asymptotic central limit theorem for two-layer random-init networks, proving convergence of the random function law to a Gaussian process… | ||
| Is Overfitting Actually Benign? On the Consistency of Interpolating Methods | Preetum Nakkiran | 26m |
Claims that modern overparameterized interpolating methods fail consistency: training MLPs to interpolation on CIFAR with controlled label corruption and on… | ||
| On the Cryptographic Hardness of Learning One-Hidden Layer Neural Networks | Ilias Zadik | 19m |
Claim: learning one-hidden-layer ReLU networks with Gaussian inputs and polynomial width is provably hard to do better than chance in polynomial time once each… | ||
| Representation Costs of Linear Neural Networks: Analysis and Design | Mina Karzand | 36m |
Derives closed forms for linear networks: fully connected depth D gives representation cost proportional to D·sum σ_i^{2/D} (σ_i singular values); diagonal and… | ||
| nothing matches | ||