Dynamo: Supporting Next-Generation AI Workloads
summary
Dynamo is an open-source, Rust-native inference framework with Python bindings that targets multi-node generative AI serving and ships with a separate data-plane library called Nixel for RDMA/TCP transfers. It implements disaggregated serving (separate prefill and decode pools), a KV cache with KV-aware smart routing, and a planner for dynamic GPU reassignment to reduce wasted generation capacity. CLI tools (dynamo run, dynamo serve) deploy graphs against TRTLM, VLM and SGLAN backends, the repo shows ~99 closed external PRs and ~32 open issues.
AI-written summary · about
chapters (15)
- 0:00Dynamo and Nvidia intro
- 1:03Generative AI inference challenges
- 6:20Distributed inference needs
- 8:27Dynamo architecture overview
- 12:00KV cache in LLMs
- 13:52GPU scheduling and routing
- 15:16Request routing strategies
- 18:28Nixel data transfer engine
- 20:19Disaggregated prefill/decode
- 22:25Dynamo deployment examples
- 24:10Multi-model in Dynamo
- 27:03Multi-node setup
- 28:29Nvidia Dynamo features
- 31:59Getting started with Dynamo
- 33:25Q&A: Dynamo vs competitors
related talks
19mDisaggregated Intelligence: Scaling AI Agents with Dynamo and Kubernetes
20mHacking the Inference Pareto Frontier
13mDeploying Enterprise-Grade Compliant Generative AI
28mScaling agentic AI: From infrastructure to enterprise outcomes
19mScalable Data and AI Solutions with Cloud-Native Architecture
24mKeynote: Next-generation frameworks for Large-scale AI
26mEfficiently serving LLMs at scale.- 28mScaling agentic AI: From infrastructure to enterprise outcomes – Kevin Cochrane