ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native...
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
ECHO offloads KV caches to host memory, expanding the effective GPU K pool from ~330K to 1.8M tokens and delivering up to 2.1× higher generation throughput on long-context workloads. It implements a graph-friendly GPU-side cache manager with fixed-length metadata and parallel allocate/evict primitives, a lossless intra-query prefetch via EMA-predicted score threshold, and fused pipelined GPU kernels that overlap index computation with host→GPU recalls. Measured GPU hit rates reach 97–99%; TTFT rises up to 8% and end-to-end latency increases stay within ~10% under target loads.
AI-written summary · about
chapters (12)
- 0:00ECHO introduction
- 2:03Sparse attention and DeepSeek DSA
- 3:06HBM bottleneck in sparse attention
- 3:48Dynamic KV cache offloading
- 5:13Lossless prefetching design
- 7:41Prefetching decoding with EMA
- 8:45Prefetching prefill with top K
- 10:11Fused kernels in ECHO
- 11:18Evaluation setup and results
- 12:42Hit rate and prefetching gains
- 13:46Generality and related work
- 14:07Summary
related talks
15mKV Cache: Accelerating AI Inference on Intel CPU
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
19mOffloading-Efficient Sparse AI Systems
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
14mSYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
26mEfficiently serving LLMs at scale.
44mVLLM K/V Caching With Ceph