Beyond the Theory: What Actually Breaks When You Scale Your Disaggregat...
summary
Disaggregated inference runs vLLM and SGLang on Kubernetes with Dynamo or LLMD, Kai for scheduling, and Grove or leader-worker-set for workload APIs, and surfaces five concrete failure modes: gang scheduling across components and within components to avoid GPU deadlocks, KV-cache transfers kill latency when prefill and decode land in different topology zones, partial allocations waste GPUs, and standard autoscalers misbalance prefill/decode SLAs; Grove plus Kai proves the multi-level gang/topology model works better than naive deployments.
AI-written summary · about
chapters (15)
- 0:00Disaggregated serving challenges
- 1:03Why disaggregated serving exists
- 4:12Five failure modes scaling models
- 7:24Resource deadlocks in gang scheduling
- 8:48Topology-aware placement for latency
- 10:55Production-ready disaggregated serving
- 11:59Kubernetes layers for inference
- 12:41Kai scheduler for GPU allocation
- 13:22Leader Worker Set for multi-host
- 14:51Solving partial allocations with Kai
- 15:55Topology-aware scheduling
- 16:57Introducing Grove for gang scheduling
- 18:27Key takeaways for inference deployment
- 19:51Choosing tools by use case
- 20:11Open source projects and resources
related talks
39mBeyond Model Sharding: Atomic Scheduling and Disaggregated LLM Serving With ...
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
01h 03mScaling up machine learning without tears (and what do programming languages ha...
32mScaling Generative AI Inference with llm-d
38mBeyond Kubernetes: Adapting To Spe...
27mLive Q&A Session 1
11mEngineering Scalable ML Infrastructure for Modern Apps
29mOptimizing Training Performance for Large Language Model(LLM) in Kubernetes