Serving Frontier MoE Models at the Lowest Token Cost
summary
A 72‑GPU rack called GB300 NVL72 is built with 18 trays and nine NVLink switches to give each GPU 1,800 GB/s and an all‑to‑all fabric of 130 TB/s, eliminating MoE dispatch/combine bandwidth bottlenecks. Benchmarks claim 20x more performance on agentic workloads vs an 8‑GPU Hopper node and tokens delivered at a tenth of the cost; NVIDIA Dynamo adds up to 7x software speedup and Ray Serve adds NVLink group placement while Vera Rubin pod claims a further 30x uplift.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 1:11Model growth challenges
- 3:18Mixture of Experts costs
- 5:25GB300 NVL72 solution
- 6:52NVIDIA software stack
- 9:00Open-source collaboration
- 9:45Performance gains with GB300
- 10:49Cost efficiency benefits
- 11:30NVIDIA Dynamo integration
- 12:55Ray NVLink awareness
- 13:15Future with Vera Rubin
related talks
20mHacking the Inference Pareto Frontier
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
30mBeyond the DC Walls: Building a Nationwide GPU Fabric With KubeVirt and Wide-Area…
31mThe Evolution of Multi-GPU Inference in vLLM
29mBuild Customize and Deploy LLMs At-Scale on Azure with NVIDIA NeMo
22mPerformance Optimizations at 100K+ Scale
21mAnalystANGLE
28mHarnessing generative AI with NVIDIA AI and Microsoft Azure