Specializing Small Language Models with less data
summary
Claims that large models synthesize labeled data to fine-tune small models, using knowledge distillation and an API-driven pipeline that validates generated examples, runs teacher evaluation, then trains a student on 50 examples plus unlabeled context. Empirical result: a fine-tuned 1 billion model or an 8B Llama variant matches a 400B teacher (teacher called lava 405b) on extractive QA benchmarks; pipeline outputs downloadable binaries and on-device deployable models that actually reach parity.
AI-written summary · about
chapters (14)
- 0:00Specialized small language models
- 0:43AI products and multi-agent systems
- 2:29Expert agents and their stack
- 3:54Small language models efficiency
- 5:01Building expert agents challenges
- 6:24Knowledge distillation for agents
- 8:31Input classification workflow
- 10:19Fine-tuned vs LLM benchmarks
- 12:04Live model training demo
- 13:11Model training data needs
- 16:04Teacher evaluation process
- 17:53Base vs fine-tuned performance
- 20:02Key takeaways and deployment
- 21:06Closing and resources
related talks
08mLeveraging large-scale models for efficient learning
55mNo Data? No Problem: Zero-Data Model Training Foundational Models
18mSmolLM: The Rise of Smol Models
16mExploring the world of Small Language Models
28mFine-Tuning Small Language Models with Python
30mLearning to wield large language models
01h 38mTraining Large Language Models: Practices and Research Questions
38mMeta's Roadmap for Full Stack AI