Long context modeling and generalization: two perspectives
summary
Models show perplexity exploding past their pretraining length (a 4k-pretrained model becomes nearly unusable by ~5.5–6k tokens). The work builds a retrieval-at-attention encoder (example: Unlimiformer) and cites causal-model adaptations (LM-Infinite, InfLLM) that keep recent and initial tokens and top-k middle retrieval. Positional RoPE extrapolates poorly but NTK-aware/ABF scaling helps. Empirically, long-context ICL fits up to ~2,000 demos (on datasets like Clinic-150) with retrieval gains shrinking to ~2.5 points at 1,000 demos; adversarial label sorting ruins performance, while local blocks of ~50–100 examples recover near-full accuracy.
AI-written summary · about
chapters (19)
- 0:00Start
- 0:48Introduction
- 1:49Long context modeling challenges
- 2:51Three length generalization strategies
- 6:49Model adjustments for generalization
- 10:20Unlimiformer and attention retrieval
- 13:40Retrieval-based approach limitations
- 14:23RoPE and positional embeddings limits
- 18:39Positional embedding modifications
- 18:59Long-context in-context learning
- 21:52Long-context ICL trade-offs
- 28:21ICL demo ordering sensitivity
- 34:29Adversarial label sorting
- 36:41Local attention experiments
- 41:26Long-context ICL key findings
- 42:48Long context solutions and limits
- 43:51Evaluating long context
- 48:09Defining long context and future work
- 50:22Q&A
related talks
01h 00mThe Frontier between Retrieval-augmented and Long-context Language Models
15mStop Stuffing Your LLM's Context Window
49mContext is King
30mLarge Language Models and Prompt Engineering for Developers
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
01h 16mGeneralization in natural language processing
01h 30mToward Understanding In-context Learning
01h 38mTraining Large Language Models: Practices and Research Questions