DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants
summary
DroidSpeak enables KV cache reuse across fine‑tuned variants of the same architecture by profiling model pairs to find a small set of critical layers (about 11% of layers on average) and recomputing only those while reusing the rest; it transmits hidden states to start layer‑wise recomputation pipelined with cache transfer. Deployed as a distributed inference system across nodes, it reports up to 4× throughput and about 3.1× faster prefill with negligible loss in F1/Rouge‑L/code similarity.
AI-written summary · about
chapters (8)
- 0:00DroidSpeak introduction
- 1:23Repeated context processing
- 3:51KV cache sharing limits
- 6:41KV cache translator design
- 9:10Critical layers for recomputation
- 12:20System optimization pipeline
- 14:10Evaluation setup and results
- 16:20Conclusion and future
related talks
22mCost-Efficient Large Language Model Serving for Multi-turn Conversations with...
17mApproximate Caching for Efficiently Serving Text-to-Image Diffusion Models
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
15mKV Cache: Accelerating AI Inference on Intel CPU
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe