Human and machine inductive biases for compositional linguistic generalization
summary
Benchmarks COGS and SLOG test mapping sentences to logical forms and show models reliably solve lexical substitution but fail most structural generalization unless given explicit structural scaffolding. Meta-analysis finds many seq2seq/LMs reach high lexical accuracy; Llama fine‑tuned on SLOG scores ~40%. Pipeline solutions that inject syntax (iterative parsing + prompting) reach near 100%, while opaque pretraining can inflate reported success by ~14–19 percentage points. Human artificial‑language tests show most humans generalize modifiers to novel positions.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:10Compositional linguistic generalization
- 6:49Human and machine inductive biases
- 9:57Systematicity debate challenges
- 11:24Lexical versus structural generalization
- 15:41Human structural generalization evidence
- 23:35Human-like generalization in AI benchmarks
- 26:27Frequency gaps in benchmarks
- 29:36Neural networks and lexical generalization
- 34:20Structural generalization difficulties
- 40:56Scalable compositional solutions
- 43:50Opaque pre-training data challenges
- 46:46Generalization in semantic parsing
- 48:30Human structural generalization gaps
- 49:12Summary and future directions
- 50:20Q&A
related talks
01h 16mGeneralization in natural language processing
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between
26mTowards controllable language generation
45mReasoning in Natural Language
55mLessons Learned From Implementing Generative AI Solutions
01h 31mNatural Language Understanding: Foundations and State-of-the-Art
01h 18mThe Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes
01h 03mLLM skills and meta-cognition: scaffolding for new forms of learning?