Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional...
summary
Bidaw implements a two-tier KV cache (host memory + SSD) and coordinates compute and storage: IO-aware request scheduling separates and reorders requests by where KVs live and by KV size to avoid GPU blocking, while answer-length guided eviction uses generated response length to predict reuse and keep hot KVs in memory. It also applies storage-efficient tensor caching to trade space for compute. On interactive workloads Bidaw cuts latency up to 3.58×, boosts throughput up to 1.83×, and lowers P90–P99 latencies substantially with sub-millisecond per-operation overhead.
AI-written summary · about
chapters (10)
- 0:00Bidaw introduction
- 1:02KV caching challenges
- 3:33Two-tier storage access
- 6:29Bidirectional compute-storage
- 7:53IO-aware scheduling
- 9:19Conventional scheduling comparison
- 10:00Answer-based KV eviction
- 11:48GPU memory optimizations
- 12:30Evaluation setup
- 13:59Summary
related talks
15mKV Cache: Accelerating AI Inference on Intel CPU
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
16mNo Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
44mVLLM K/V Caching With Ceph
26mEfficiently serving LLMs at scale.
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
33mScaling KV Caches for LLMs: How LMCache + NIXL Handle Network and Storage...