Understanding Generalization from Pre-training Loss to Downstream Tasks
summary
Contrastive pretraining provably produces embeddings where contrastive loss → spectral clustering and Euclidean distance ≈ manifold distance on the population positive‑pair graph, so downstream labels that are constant on graph clusters become linearly separable (sample complexity scales with the number of implementable clusters). The work formalizes model‑implementable clusters (r0) versus raw cluster count r and shows the need for architecture inductive bias, and it documents implicit bias of optimizers (flatter minima) affecting transfer both empirically and theoretically.
AI-written summary · about
chapters (10)
- 0:00Introduction to pre-training
- 4:19Theoretical motivations
- 10:51Contrastive learning basics
- 13:43Graph-based interpretation
- 24:42Spectral clustering connection
- 34:00Linear separability proof
- 37:55Empirical vs population graphs
- 47:45Transferability and embeddings
- 53:52Architecture and optimization
- 59:01Model implementability and wrap-up
related talks
45mUnderstanding Contrastive Learning and Self-training
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
01h 02mUnexpected Test Losses from Generalization Theory?
38mRepresentation via Representations: Improving Generalization to Unseen Populations
01h 01mDistribution Shift as Underspecification, and What We Might Do About It
59mSize (OOD) Generalization of Neural Models via Algorithmic Alignment
37mA Bluffer's Guide to Dimension Reduction