Performance Optimizations at 100K+ Scale
summary
Meta scales Llama 4 training to nearly two trillion parameters on over 100k GPUs by reworking topology (oversubscription cut from 1:7 to ~1:2.7 across buildings), forking NVIDIA’s nickel into NikoX with a custom transport CRM, moving to zero copy RDMA and a host-driven algorithm model, and adding DQPLB topology-aware queue load balancing. Results claim 20–50% better collective benchmarks, ~10% end-to-end model speedups and >15% overall training efficiency gains.
AI-written summary · about
chapters (11)
- 0:00Meta’s AI cluster performance
- 1:29Scaling network at 100K+ GPUs
- 3:54Mapping parallelisms to network
- 5:20Evolving collective communication
- 8:52Zero-copy for GPU efficiency
- 10:59Host-driven algorithm changes
- 14:11Optimizing inner-domain communication
- 15:41Custom transport with dynamic QA load balancing
- 17:45Benchmark results and topology
- 19:13Future directions in model and hardware
- 20:42Key takeaways from Llama 4
related talks
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3
22mGenAI Training In Production: Software, Hardware & Network Considerations
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
48mBringing Llama 3 to Life
31mBuilding LLaMA: Meta's Director of GenAI
20mHacking the Inference Pareto Frontier
26mEfficiently serving LLMs at scale.