Leveraging large-scale models for efficient learning
summary
Large pretrained models guide smaller ones by matching the teacher’s next-token probability distributions rather than hard one-hot targets, which speeds convergence and cuts required data and compute; experiments report a 2B model showed consistent gains on reasoning, code and common-sense benchmarks after distillation. Requires a high-quality teacher (Gemma or a 100B Llama-style model), so teacher must be strictly better and student performance is upper-bounded by teacher capacity, with risks of teacher leakage / test overfitting.
AI-written summary · about
chapters (5)
- 0:00Strategies to improve LLMs
- 2:10Fundamentals of training
- 3:52The concept of distillation
- 4:35Benefits and challenges
- 7:50Closing remarks
related talks
22mSpecializing Small Language Models with less data
18mImproving open language models at a practical size
01h 38mTraining Large Language Models: Practices and Research Questions
44mTransfer learning in NLP
21mEfficientBERT: Trading off Model Size and Performance
28mFine-Tuning Small Language Models with Python
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason
18mHow to Make Small Language Models Work