Bringing Llama 3 to Life
summary
Meta releases an open-weight Llama 3 family including a 405B / 45B model trained on about 15 trillion tokens using 16k H100 GPUs, with the 45B comparing well on 150+ benchmarks and ~2k human prompts. Data pipeline upgrades include a 128k tokenizer, synthetic-data rejection sampling with code execution checks, and extensive deduplication. Training runs last weeks (54 days for a major job), hit 419 incidents (78% hardware), yet achieve ~90% effective training time.
AI-written summary · about
chapters (5)
- 0:00Overview of Llamas
- 9:09Data for GenAI
- 20:03Llama Training at Scale
- 34:06Llama Inference at Meta
- 45:48Closing
related talks
31mBuilding LLaMA: Meta's Director of GenAI
38mMeta's Roadmap for Full Stack AI
17mThe State of the Llama Ecosystem
12mUnpack the Latest Llama 4 Herd
03mFinetune Llama 4 Faster
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
12mKeynote: The Llama Ecosystem: Past, Present and Future
29mLeveraging open-source LLMs for production