Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture...
summary
Mooncake builds a KVCache-centric global cache and disaggregates prefill and decode so prefill computes once and decoding pulls reusable KVCache blocks from Monstore over GPU‑direct RDMA/TCP/MME, saving GPU flops at the cost of SSD/DRAM/CPU storage. Real deployment handles thousands of nodes, >100B tokens/day, cuts system cost by >50%, and boosts effective request capacity by 59%–498% versus baselines, letting Kimi serve 115% and 107% more requests on A800 and H800 clusters; KVCache for 1M tokens can reach ~320 GB.
AI-written summary · about
chapters (11)
- 0:00Introduction to Mooncake
- 0:54LLM inference challenges
- 2:13System requirements and goals
- 3:54Prefix caching analysis
- 6:17Mooncake system architecture
- 7:12Disaggregated inference design
- 8:41Mooncake store infrastructure
- 10:28Transfer engine and integration
- 12:01Performance evaluation
- 14:02Ablation and future work
- 15:22Conclusion and summary
related talks
12mKV Cache and Long-Context Inference: Solutions for Inference at Scale
26mLMCache: Lower LLM Performance Costs in the Enterprise
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
26mEfficiently serving LLMs at scale.
12mVertical Mobility: Inference from MVP to Trillion Parameters
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
40mScalable and Efficient LLM Serving With the VLLM Production Stack
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe