Learning from multiple modalities, Predicting on unseen tasks
summary
Zero-shot prediction claims to deliver strong downstream accuracy with no task-specific labels by training large contrastive or self-supervised models on image–text pairs (CLIP, DINO) and then scoring class pseudo-captions against image embeddings. Theory frames this either as learning a similarity score approximating the Radon-Nikodym derivative (hard pretrain, cheap downstream) or as composing conditional-mean maps, and predicts a fast prompt error scaling like 1/M prompt rate when M (often 50–100) diverse prompts are used.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:07Foundation models and zero-shot
- 4:17What is foundation modeling
- 8:33Why foundation models work
- 10:05Statistical learning theory
- 13:06Origins of zero-shot
- 15:42Pre-training vs evaluation
- 18:32Zero-shot vs supervised
- 23:05Two approaches to foundation
- 27:22Information density vs mean
- 31:13Error in zero-shot learning
- 32:50Prompt complexity and bias
- 35:15Composition rules for generalization
- 36:26Future directions
- 40:13Q&A
related talks
28mBeyond Supervised Learning with Zero-Shot Vision
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
13mInside Multimodal Models
01h 04mFrom Pixels to Words: Exploring Multi-Modal AI with Python
01h 30mToward Understanding In-context Learning
41mSemantic Multimodal Multilingual Similarity Engine
26mFew-Shot Learning in Conversational AI
31mKeynote Q&A – Barbara Plank