Unveiling the Journey of NLP
summary
Claims concrete wins and clear limits: Transformer (2017) delivers parallel attention that captures long dependencies, pretraining plus fine‑tuning lifts SQuAD by about 1.5–5.1 points, and GPT‑3: 175B params shows scaling enables zero/one/few‑shot tasks at a $4.6M training scale. Tools named and used are open‑source pretrained models, fine‑tuning, RAG (retrieval‑augmented generation) to avoid stale knowledge, and RLHF for aligned responses. Demonstrations include rapid prototyping of an email‑reply model versus a four‑month build, examples of translation working in the field, and agents that search, draft and act.
AI-written summary · about
chapters (18)
- 0:00NLP and language models
- 1:05What is a language model
- 2:29NLP’s explosion factors
- 3:14Transformer architecture
- 4:59Pre-trained models and transfer learning
- 6:02Masked language models
- 8:11Pre-trained models impact
- 9:17Open source models
- 10:20GPT-3 breakthroughs
- 11:26Few-shot learning methods
- 13:13Fast prototyping with GPT
- 14:16ChatGPT and reinforcement learning from human feedback
- 15:57AI trends shaping future
- 17:00Retrieval augmented generation
- 17:43AI agents acting on our behalf
- 19:08Multilingual model breakthroughs
- 20:38Multimodality beyond text
- 21:42Summary and future directions
related talks
40mPanel discussion "NLP"
01h 01mKeynote: Transformers: The Rise of ChatGPT
44mTransfer learning in NLP
28mFine-Tuning Small Language Models with Python
01h 38mTraining Large Language Models: Practices and Research Questions
45mTransfer Learning - Entering a new era in NLP
29mHow AI Models Get Smarter
01h 16mGeneralization in natural language processing