Distribution Shift as Underspecification, and What We Might Do About It
summary
Proposes DivDis which trains multiple hypotheses by minimizing training loss while maximizing disagreement on unlabeled target examples then disambiguates with a few labeled queries Results: often only two heads suffice, 16 active queries choose the correct head and DivDis improves worst‑group accuracy by >25% on three of four constructed complete‑correlation datasets outperforms baselines tuned without group labels generalizes on WILDS Chameleon domain shift also applies to RL via a latent z structured maxent method that yields better five‑shot adaptation than adversarial or diversity‑only baselines and shows that invariances fail to transfer to long tail classes unless explicitly encouraged
AI-written summary · about
chapters (17)
- 0:00Introduction to shift
- 1:21Real-world shift examples
- 2:58Data and datasets
- 4:43Handling distribution shift
- 7:04Limitations of methods
- 8:21Under-specified problems
- 13:21Diverse predictions
- 17:37Model disambiguation
- 20:38Experiments and results
- 26:21Robustness benchmarks
- 28:51Medical and RL settings
- 32:38Technical Q&A
- 37:41Linearity and ensembles
- 43:23RL for diverse solutions
- 47:30Controllable diversity
- 52:17Visualizing RL results
- 55:34Label shift and invariance
related talks
50mRobust Deep Learning Under Distribution Shift
42mBeyond Adversarial Approaches on Distributional Robustness
41mDistribution shift in ecological data: generalization vs. specialization,
37mDomain Adaptation-- 20 years of theory chasing practice
54mA Discrepancy-Based Theory of Adaptation
54mLearning Uninformative Representations
48mA New Paradigm for Learning with Distribution Shift
52mLanguage-guided Adaptation