CacheSlide: Unlocking Cross Position-Aware KV Cache Reuse for Accelerating LLM Serving
summary
Describes CacheSlide, a KV-cache management system for LLM serving that targets agent-style prompts with invariant and dynamic segments. It identifies a Relative-Position-Dependent Caching (RPDC) pattern and implements Chunked Contextual Position Encoding plus Weighted Correction Attention to reuse cached key/value tensors across shifted absolute positions, only recomputing attention for a small subset of tokens and adding layer-wise and spill-aware cache optimizations. The prototype extends vLLM and reports 3.11–4.3× latency reductions and 3.5–5.8× throughput gains over prior Position-Dependent and Position-Independent caching baselines.
AI-written summary · about
related talks
34mVLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
30mDesigning Fast & Cost-Efficient Agents: The Power of Caching
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
15mKV Cache: Accelerating AI Inference on Intel CPU
16mReinforcement Learning-Based SLC Cache Technique for Enhancing SSD Write
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...