Learning Uninformative Representations

on YouTube ↗54m

" summary

Claims an algorithm called EIEL that infers worst-case domains from a reference model that has learned shortcuts, then alternates environment assignment (Bernoulli Q, inner loop maximizing an IRM-style proxy) with invariant training (outer loop). Tools include adversarial objectives, MMD references and the IRM gradient proxy. Tests on ColorMNIST (ERM drops to ~10% when colors flip; invariant methods reach ~70%) and Waterbirds show improved worst-group accuracy, but success depends on the reference model finding spurious features.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 0:38Removing Information from Representations Long-standing aim of representation learning: preserve information in data
3. 1:46Info Removal: Remove spurious features Aim to learn predictor, factor out particular known spurious features Example: diabetes predictors, uncovering factors beyond BMI
4. 2:52Info Removal: Information Bottleneck .
5. 3:31Info Removal: Self-Supervised Learning
6. 4:31Info Removal: Invariant learning
7. 6:24Illustration of Invariant Learning
8. 8:30Info Removal: Fair representation learning
9. 10:21Info Removal Methods: Distribution Matching
10. 12:39Info Removal Methods: Adversarial
11. 15:03Invariant Learning with Unknown Environment
12. 17:24Notation & Definitions
13. 19:27Invariant Learning Fairness Mary parallels between Invariant learning and algorithmic fairness
14. 30:55EIIL: Dependence on reference model
15. 33:09Discussion & Open Questions

" related talks

• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 18mRectifying Privacy and Efficacy Measurements in Machine Unlearning...Weiran Liu, Nima Naderloui, Shenao Yan, Binghui Wang, Jie Fu, Wendy Hui Wang, Yuan Hong · USENIX Security '25
• 38mRepresentation via Representations: Improving Generalization to Unseen PopulationsFrances Ding · Recent Developments in Research on Fairness 2019
• 01h 08mUnwanted Learning: What It Is, Why It Matters, and What To Do About ItVitaly Shmatikov · Beyond Differential Privacy 2019
• 31mIndividual Gaps in Prediction GeneralizationMarzyeh Ghassemi · Bridging Prediction and Intervention Problems in Social Systems (Simons Institute, 2026)
• 45mTBDKamalika Chaudhuri · Domain Adaptation and Related Areas 2024
• 01h 12mRecent Progress in High-Dimensional LearningAnkur Moitra · Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020)
• 49mOracle-Efficient Online Learning or: How to Use Non-Robust Optimization for Robust LearningNika Haghtalab · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond