KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a...
summary
Paper analyzes a one-week Alibaba trace with per-request timestamps, modalities, turns and KV hashes and finds reuse is skewed: single-turn requests contribute substantial hits, most KV blocks are ephemeral (90% unused after 10 minutes in trace A, 0.3s in trace B), and ideal cache size is moderate. It proposes a workload-aware eviction that fits per-workload exponential reuse and a short-life window, improving hit rate by 1–24% and cutting end-to-end TTF by up to 42%; traces are open-sourced.
AI-written summary · about
chapters (10)
- 0:00KVCache cache management
- 1:51Design goals for KVCache
- 3:18KVCache locality challenges
- 5:46Limitations of workload-agnostic caching
- 7:10Workload-driven KVCache approach
- 8:55Trace analysis methodology
- 9:38Key insights from workload characterization
- 11:26Ephemeral KVCache reuses
- 13:14Workload-aware eviction policies
- 15:40Conclusion and future work
related talks
28mCache for Cash—Speeding Up Production with Kafka and MySQL binlog
17mPushing the Limits of In-Network Caching for Key-Value Stores
20mAdaptive Online Cache Capacity Optimization via Lightweight Working Set Size...
42mCache, But Cache Wisely!
19mCaching with Cache2K
13mDifferentiated Key-Value Storage Management for Balanced I/O Performance
33mCache is King: Get the Most Bang for Your Buck From Ruby
32mKeyvi the Key Value Index Optimized for Size and Speed #bbuzz