SYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems
summary
SYMPHONY builds a low-latency disaggregated memory layer that moves KV caches off GPUs using advisory requests prefetch and a multi-tier storage pipeline, keeping early-layer KV state on GPU via priority-based KV cache allocation and coordinating frees with the serving engine through cooperative memory management. Evaluated on LLaMA models with ShareGPT and agent workloads, it cuts end-to-end latency 2.4× vs vLLM, serves 4× more requests, achieves 5× faster time-to-first-token vs Loomnex+Infrasept, and tolerates spurious hints with up to 36% max-latency increase.
AI-written summary · about
chapters (10)
- 0:00Symphony intro and compute-memory disagg
- 3:05Stateful LLM serving challenges
- 4:29Low-latency memory needs
- 5:53Prefetching with unreliable hints
- 7:39Symphony architecture
- 8:21Request serving with memory mgmt
- 10:30Evaluation setup
- 11:55Performance improvements
- 13:21Handling spurious hints
- 13:42Conclusion and future
related talks
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
26mLMCache: Lower LLM Performance Costs in the Enterprise
40mScalable and Efficient LLM Serving With the VLLM Production Stack
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
19mSolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…