IMPRESS: An Importance-Informed Multi-Tier Prefix KV Storage System for Large Language...
summary
IMPRESS is a storage stack that cuts time-to-first-token by up to 2.8× versus state-of-the-art while holding inference accuracy within ~0.2% of full recomputation. It runs on an A100 server (80 GB GPU, 128 GB CPU, 2 TB SSD) and tests 55–65 GB prefix datasets with a 10 GB GPU + 32 GB CPU cache and 64-token chunks. Key mechanisms are similarity-guided important-token identification that loads keys from a few heads and approximates others, KV reordering to separate important chunks and reduce read amplification, and score-based cache admission that prefers importance over raw frequency.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 1:07Prefix KV storage bottleneck
- 2:29Challenges in existing systems
- 3:09The IMPRESS architecture
- 10:50Experimental results
- 12:44Conclusion and Q&A
related talks
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
38mOptimizing LLM Inference for the Rest of Us
22mCost-Efficient Large Language Model Serving for Multi-turn Conversations with...
26mEfficiently serving LLMs at scale.
12mKV Cache and Long-Context Inference: Solutions for Inference at Scale
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference