Beyond the Node: Scaling Inference with Cluster-Wide KVCache...
summary
Claims a cluster-wide KVCache that turns prefill recomputation into cheap fetches, demonstrated on Llama 8B with KVCache offloaded to shared DRAM and served over a custom Rust RDMA library. Benchmarks show about 11x faster Time To First Token versus full prefill and similar order-of-magnitude throughput gains, with experiments across multiple instances on a single node and a path to scale to thousands of machines storing terabytes of cache.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:41Scaling AI inference beyond single nodes
- 2:27What is KV cache and why it matters
- 3:31Key use cases for KV cache
- 6:03Shared KV cache benefits
- 7:05Offloading KV cache to DRAM
- 8:09Disaggregated inference frameworks
- 10:21Kubernetes-like KV cache management
- 12:06Vertically integrated KV cache system
- 13:11Single-node performance gains
- 14:16Shared KV cache across models
- 16:45Remote KV cache fetching with RDMA
- 17:30Scaling KV cache to thousands
- 18:13Conclusion
related talks
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
15mKV Cache: Accelerating AI Inference on Intel CPU
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
32mScaling Generative AI Inference with llm-d
44mVLLM K/V Caching With Ceph
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
15mECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native...