How to Make Small Language Models Work
summary
Starts from a tiny GPT-2-style teacher and overgenerates outputs, uses aggressive three-stage filtering (including a PMI-style information-theoretic filter) to curate training data, and distills repeatedly into a 0.5B student that matches or outperforms GPT-3.5 on document summarization depending on eval. Builds an n-gram service called infin using a suffix array index (indexed for a few hundred dollars, served 60M API calls, millisecond latencies) and shows interpolation with neural models lowers perplexity.
AI-written summary · about
chapters (6)
- 0:00Impossible possibilities
- 0:57Data synthesis advantages
- 3:37Distillation workflow
- 7:15Competing with GPT-3.5
- 10:31The Infinigram project
- 15:18Future of AI data
related talks
44mHow to build your own Tiny Language Model from scratch
28mFine-Tuning Small Language Models with Python
39mCollect Data to Fine-Tune Your Large Language Model
01h 00mInside GPT – Large Language Models Demystified
51mLarge Language Models for Real-World Applications - A Gentle Intro
28mGPT or how I learned to stop worrying and love the generative AI
43mBeyond the Hype: A Realistic Look at Large Language Models
23mUnveiling the Journey of NLP