Generalization in natural language processing
summary
Survey of what NLP actually generalizes to: charts the shift from lexicon-plus-grammar pipelines to large self‑supervised models, shows transformers trained on web data plus RLHF and few‑shot prompting produce many task behaviors, cites Gemini 1.5 Pro’s 10 million tokens in context claim and BLIP-style image‑alt denoising, reports code‑T5 syntax‑aware masks improve code generation, notes interpretability probes beat attention importance, demonstrates iterated reference games fail to induce new conventions, and uses the apricate example to expose the long‑tail evaluation gap.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:23Talk purpose and scope
- 16:38Historical NLP tasks
- 25:00NLP paradigm shift after 2019
- 38:59What is language
- 51:03Language as conventions
- 1:03:16Long tail in language modeling
- 1:11:22Implications for language tech
- 1:15:15Closing and Q&A
related talks
23mUnveiling the Journey of NLP
40mPanel discussion "NLP"
01h 00mHuman and machine inductive biases for compositional linguistic generalization
17mNatural Language Processing in Automation
06mAdvancements in Natural Language Processing (NLP) and Text Analytics
43mBeyond the Hype: A Realistic Look at Large Language Models
45mVenturing out into the world with AI
01h 31mNatural Language Understanding: Foundations and State-of-the-Art