Scaling Llama4 Training to 100K
summary
Claims scaling Llama4 training to a 100k‑GPU class cluster and raising effective GPU utilization from ~50% to ~90%. Key wins: replacing root init with TCP store cuts ring formation from ~132s to ~10s at 96k, a bidirectional ring all‑gather halves metadata exchange, and CPU emulation of GPU collectives exposes an n² CPU hotspot and a 64k listen‑queue limit. Built a fault‑tolerant all‑reduce with coordinator-driven shrink/grow that contains reconfigure to ~3–4 minutes and resumes full-sync training with replicas.
AI-written summary · about
chapters (6)
- 0:00Scaling challenges for LLMs
- 2:32Analyzing initialization time
- 4:41Collective optimization
- 9:09Emulation and scaling hurdles
- 14:04Fault tolerant training
- 19:18Learnings and future outlook
related talks
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
22mGenAI Training In Production: Software, Hardware & Network Considerations
48mBringing Llama 3 to Life
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
40mScalable and Efficient LLM Serving With the VLLM Production Stack
42mStop Treating LLMs Like REST APIs