Language-guided Adaptation
summary
Language acts as explicit advice to fix domain shift: adding an attention loss from Grad-CAM/RISE to force models to be right for the right reasons; using CLIP semantic-vector edits to create embedding augmentations; and conditioning diffusion models to generate pixel augmentations. The three systems GALS LADS ALIA report consistent gains (often ~10–20 points versus zero-shot CLIP), sometimes matching extra real images but requiring outlier rejection. A follow-up set-difference captioning (VisDiff) pipeline uses captioning plus CLIP ranking to surface concrete dataset or model differences automatically.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:06Talk overview
- 2:52Language as domain shift guidance
- 8:22GALS paper
- 19:30LADS paper
- 28:55ALIA paper
- 39:01Set Difference Captioning
- 47:14ImageNetV1 vs V2 analysis
- 49:11Q&A
related talks
13mInside Multimodal Models
45mScale Learning and Reasoning Across Heterogeneous Gradients and Semantics
01h 00mHuman and machine inductive biases for compositional linguistic generalization
32mLinguistic, domain and task adaptation of LLMs using the PLLuM models as an example
32mEverything you wanted to know about VLMs but were afraid to ask
55mDeep Emergent Communication: How It Started and How It’s Going
41mLearning from multiple modalities, Predicting on unseen tasks
30mHow to train a large diffusion model