Hacking the Inference Pareto Frontier
summary
NVIDIA Dynamo is an open source datacenter inference framework that combines disaggregation, KV routing and inference-time scaling to reshape the cost/latency/quality Pareto frontier. On 16 H100s with Llama‑7B, disaggregation doubles tokens/sec/GPU at fixed latency; re-querying a small model 3–4× matches quality of a 49B model; offloading KV during predictable tool calls preserves cache across 30s waits. The repo and meetup give implementation details and deployment configs.
AI-written summary · about
chapters (15)
- 0:00Introduction to Breaking the Inference Pareto Frontier
- 0:33Introduction of Kyle Cranon and NVIDIA Dynamo
- 1:31The Three Pillars of Deployment (Quality, Latency, Cost)
- 2:11Understanding the Pareto Frontier
- 3:06Application-Specific Prioritization of Quality, Latency, and Cost
- 4:32Common Techniques to Manipulate the Pareto Frontier (Quantization, RAG, Reasoning)
- 5:19Compounding Techniques
- 6:04Three Drivers for Modifying the Pareto Frontier (Scale, Structure, Dynamism)
- 6:20Scale: Disaggregation
- 11:02Scale: Routing
- 13:00Structure: Inference Time Scaling
- 16:14Structure: KV Manipulation
- 17:43Dynamism: Worker Specialization
- 18:42Dynamism: Dynamic Load Balancing
- 19:55Conclusion and NVIDIA Dynamo Resources
related talks
40mDynamo: Supporting Next-Generation AI Workloads
22mPerformance Optimizations at 100K+ Scale
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
33mScalable LLM Inference on Kubernetes With NVIDIA NIMS, LangChain, Milvus and Flu...
26mEfficiently serving LLMs at scale.
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
13mTailor-Made Inference at Scale