Robust Deep Learning Under Distribution Shift

on YouTube ↗50m

" summary

Proposes a practical pipeline that detects and corrects label shift by using an existing classifier as a low‑dimensional probe: estimate an invertible confusion matrix on source data, solve a linear system to recover the target label marginal from the mean softmax outputs, and then apply importance-weighted retraining or plug‑in reweighting. Shows a consistent estimator with error bounds depending on confusion‑matrix condition number and sample sizes, finds detection works with ~100s–1Ks examples, and warns domain‑adversarial methods break when class marginals change.

AI-written summary · about

" chapters (32)

1. 0:00Intro
2. 1:39Outline
3. 3:02Standard assumptions
4. 4:47Adversarial Misspellings (Char-Level Attack)
5. 6:07Curated Training Task Fail to Represent Reality
6. 7:57Feedback Loops
7. 8:47Impossibility absent assumptions
8. 11:06Detecting and correcting for label shift with black box predictors
9. 11:20Motivation 1: Pneumonia prediction
10. 12:08Epidemic
11. 13:17Motivation 2: Image Classification
12. 14:22The test-Item effect
13. 15:59Domain Adaptation - Formal Setup
14. 17:31Label Shift (aka Target Shift)
15. 18:47Contrast with Covariate Shift
16. 20:35Black Box Shift Estimation (BBSE)
17. 22:28Confusion matrices
18. 22:59Applying the label shift assumption...
19. 25:41Consistency
20. 26:06Error bound
21. 29:16Detection
22. 31:31Estimation error (MNIST)
23. 32:46Black Box Shift Correction (CIFAR10 w IW-ERM)
24. 35:34A General Pipeline for Detecting Shift
25. 39:35Non-adversarial image perturbations
26. 41:29Covariate shift + model misspecification
27. 41:53Implicit bias of SGD on linear networks w. linearly separable data
28. 43:12Impact of IW on ERM decays over MLP training
29. 45:21Weight-Invariance after 1000 epochs
30. 46:08L2 Regularization v Dropout
31. 46:53Deep DA / Domain-Adversarial Nets
32. 50:02Synthetic experiments

" related talks

• 42mBeyond Adversarial Approaches on Distributional RobustnessQi Lei · Adversarial Approaches in Machine Learning 2022
• 01h 01mDistribution Shift as Underspecification, and What We Might Do About ItChelsea Finn · Deep Learning Theory Workshop and Summer School 2022
• 48mA New Paradigm for Learning with Distribution ShiftAdam R. Klivans · Unknown Futures of Generalization 2024
• 37mDomain Adaptation-- 20 years of theory chasing practiceShai Ben-David · Domain Adaptation and Related Areas 2024
• 39mPseudo-Labeling for Covariate Shift AdaptationKaizheng Wang · Domain Adaptation and Related Areas 2024
• 33mWeak supervision: a new paradigm for unreliable dataEddie Bell · PyData London 2019
• 20mGreenfield vs. Brownfield Data LabelingJohannes Hotter · Conf42 Machine Learning 2022
• 23mHow to detect silent failures in ML modelsWojtek Kuberski · Conf42 Machine Learning 2022