Frontiers of Deep Learning 2019
2019-07-15..2019-07-18
Berkeley, United States
as the organisers put it
This workshop will feature an in-depth and comprehensive overview of the core challenges in the theory and practice of deep learning, with a particular emphasis on the four themes of the program: optimization, generalization, robustness, and generative methods. The aim is to expose the attendees to the current frontier of deep learning research, including presenting the "hot off the press" progress made by program participants, industry visitors, and other invited guests.
who it's for
deep learning theorists, machine learning researchers, statistical learning theorists, robustness and adversarial researchers, reinforcement learning researchers, applied machine learning engineers, graduate students in machine learning
topics
neural tangent kernel and infinite width; overparameterization and double descent; benign overfitting in linear models; gradient descent and max-margin dynamics; size-free generalization for convnets; spectral norm analysis of convolution operators; adversarial robustness and smoothed classifiers; evaluating defenses to adversarial examples; model-based reinforcement learning (SLBO); transfer learning for medical imaging; max-affine spline interpretations of networks
| title | author | length |
|---|---|---|
| Benign Overfitting in Linear Prediction | Peter Bartlett | 48m |
Shows that linear regression can benignly overfit: the minimum-norm interpolator (pseudoinverse solution) attains vanishing excess risk when the covariance… | ||
| Practical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with... | Tengyu Ma | 48m |
Proposes a model-based RL recipe that learns neural-network dynamics with a reward-aware loss (no squared error), plugs that model into a planner solved by… | ||
| Analyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent | Nadav Cohen | 46m |
Analyzes gradient flow and gradient descent over deep linear nets to get concrete dynamics: balancedness preserved at common initializations and an explicit… | ||
| From Classical Statistics to Modern Machine Learning | Mikhail Belkin | 50m |
Argues that modern ML succeeds because many methods interpolate the training set and still generalize: kernels and nets fit MNIST with 60% random labels yet a… | ||
| Learning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels | Yuanzhi Li | 50m |
Shows that under OPS condition (very large width, careful random init, tiny learning rate) SGD on deep ReLU nets provably converges to the neural tangent… | ||
| Size-free Generalization Bounds for Convolutional Neural Networks | Hanie Sedghi | 35m |
Claims size-free generalization bounds for deep convolutional nets that are independent of input pixel count and hidden map sizes, giving a bound linear in… | ||
| On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization | Jason Lee | 46m |
Experiment with a 50-neuron teacher and 100-neuron learner shows SGD fails on the well-specified 50-neuron model but hits zero test loss every run after… | ||
| A Primal-dual Analysis of Margin Maximization by Steepest Descent Methods | Matus Telgarsky | 42m |
Proves that steepest-descent dynamics on exponential/logistic losses drive linear predictors toward the max‑margin direction: norms diverge (roughly like log… | ||
| Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets | Rong Ge | 45m |
Paper proves that solutions satisfying dropout stability (and a stronger noise stability variant) are connected by explicit low-cost paths, constructed by… | ||
| Training on the Test Set and Other Heresies | Ben Recht | 50m |
Claims that huge models always overfit get tested by reproducing CIFAR-10 and ImageNet test sets: researchers resample Tiny Images and rerun Mechanical Turk… | ||
| Computation in Very Wide Neural Networks | Yasaman Bahri | 49m |
Claims that fully connected networks with width → ∞ map to Gaussian processes and that training dynamics linearize to a frozen kernel are supported with… | ||
| Mad Max: Affine Spline Insights into Deep Learning | Richard Baraniuk | 48m |
Argues that modern nets are compositions of max-affine spline operators, so each layer induces a convex power‑diagram partition and the whole net is a… | ||
| Splitting Gradient Descent for Incremental Learning of Neural Architectures | Qiang Liu | 52m |
Proposes a practical algorithm that grows networks by splitting neurons into two equal-weight copies when training reaches a parametric stationary point… | ||
| Provable Robustness Beyond Bound Propagation | Zico Kolter | 48m |
Proposes that randomized smoothing — classifying by majority vote over Gaussian-noised inputs — yields a provable certified L2 radius (r = sigma Phi^{-1}(p))… | ||
| A New Perspective on Adversarial Perturbations | Aleksander Madry | 49m |
Claim: adversarial examples come from non-robust features in the data that reliably predict labels but flip under tiny perturbations. Specific experiment… | ||
| 1Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers | Jerry Li | 47m |
Claims that adversarially train the smooth classifier (optimize the smoothed model’s loss, not data-augmented base loss) substantially improves certifiable L2… | ||
| Interpreting Deep Neural Networks (DNNs) | Bin Yu | 47m |
Transfer-learning from ImageNet conv layers plus simple ridge/LASSO regression predicts single-neuron V4 firing rates at state-of-the-art levels across 18… | ||
| Interpretability - now what? | Been Kim | 48m |
Shows that common saliency maps barely change when network weights are randomized and fail simple sanity checks, so a benchmark with controlled object… | ||
| Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples | Nicholas Carlini | 46m |
Evaluation shows many published defenses fail under careful white‑box attack: of nine ICLR 2018 defenses tested, two hold up and seven are broken. Defenses… | ||
| Robust Deep Learning Under Distribution Shift | Zack Lipton | 50m |
Proposes a practical pipeline that detects and corrects label shift by using an existing classifier as a low‑dimensional probe: estimate an invertible… | ||
| Using Deep Learning for Perception in Autonomous Systems: A Perspective from Control Theory | Claire Tomlin | 49m |
A convolutional model trained to predict short waypoints (ResNet-50 backbone) supervises planning and control (third-order spline planner + LQR) instead of… | ||
| Kernel and Deep Regimes in Overparameterized Learning | Suriya Gunasekar | 46m |
Gradient descent on overparameterized models converges to qualitatively different minimizers depending on parameterization and initialization scale: factorized… | ||
| Towards Understanding Transfer Learning with Applications to Medical Imaging | Maithra Raghu | 46m |
Paper "Understanding Transfer Learning for Medical Imaging" evaluates ImageNet pretraining vs random init on chest X-rays and diabetic retinopathy scans (~200k… | ||
| Meta-learning of Optimizers and Update Rules | Jascha Sohl-Dickstein | 47m |
Meta-learning replaces hand-designed optimizers with a parametric per-weight MLP optimizer (≈2k outer parameters) and faces three practical failures… | ||
| Studying Generalization in Deep Learning via PAC-Bayes | Gintare Karolina Dziugaite | 45m |
Work-in-progress applies PAC-Bayes to predict SGD generalization by building Gaussian fractional-data prior learned on a subset S' (sometimes via differential… | ||
| Sample-complexity of Estimating Convolutional and Recurrent Neural Networks | Aarti Singh | 35m |
Claims minimax sample complexity: two‑layer linear CNN with average pooling needs about M/N rate samples (filter size M), weighted pooling needs (M+J)/N, and… | ||
| nothing matches | ||