Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
EcoServe is a serving system that implements partially disaggregated (PaDG) scheduling to avoid KV-cache transfers while reducing prefill–decode interference; it builds a coordinating unit called a macro instance that staggers prefill windows across ordinary instances and uses an adaptive scheduler plus mitosis scaling for fine-grained capacity changes. On a 32‑GPU NVIDIA L20 Ethernet cluster it doubles goodput (1.96×–2.51×) for 30B–70B models versus vLLM, Sarathi, DistServe, and MoonCake, and remains competitive on H100/NVLink setups.
AI-written summary · about
chapters (11)
- 0:00Efficient LLM serving intro
- 1:02LLM inference two stages
- 2:47Non-disaggregated vs disaggregated
- 6:19Commodity GPU cluster challenges
- 7:43EcoServe partially disaggregated
- 9:10EcoServe macro scheduling
- 11:40Mitosis scaling for capacity
- 13:50Evaluation setup and baselines
- 15:13SLO and goodput results
- 16:18Scaling behavior and compatibility
- 17:44Conclusion and future
related talks
14mSYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
26mEfficiently serving LLMs at scale.
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3
35mLocal LLMs on low end hardware - a practical perspective
10mLightning Talk: Best Practices for LLM Serving with DRA