Building LLaMA: Meta's Director of GenAI
summary
Llama evolution moves from a 1.4 trillion tokens research-only pretrain (2k context) to instruction-tuned Llama 2 at 2 trillion tokens (4k context) and a Llama 3 family trained on 15 trillion tokens with up to 128k context and models like 70B and a 405B class where quantized inference still needs many H100s. Training runs use ~16,000 GPUs and suffer about eight interruptions a day. Major bottlenecks: dataset scale (would need ~150T tokens to scale 10x), system reliability, evaluation drift, and post-training data/benchmark scarcity; DPO and synthetic data are proposed mitigations.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:07Evolution of LLaMA models
- 3:59LLaMA 2 improvements
- 6:47LLaMA 3 and 4 releases
- 9:19Pre-training challenges
- 12:31Hardware and emotional challenges
- 14:59Scaling laws limitations
- 16:44Data scarcity for scaling
- 18:12Post-training complexity
- 22:07Benchmarking and community needs
- 24:58Q&A session
related talks
48mBringing Llama 3 to Life
17mThe State of the Llama Ecosystem
12mKeynote: The Llama Ecosystem: Past, Present and Future
12mUnpack the Latest Llama 4 Herd
16mLlama 4
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
30mUnderstanding the end-to-end LLM training and inference pipeline
29mLeveraging open-source LLMs for production