Cost-Efficient Large Language Model Serving for Multi-turn Conversations with...
summary
A hierarchical KV caching system called CachedAttention reuses key/value tensors across multi-turn chats by spilling caches to SSD and host DRAM, using layer-wise pre-loading and asynchronous saving plus scheduler-aware fetching/eviction to place hot caches in pinned host memory, and decoupling positional encoding so truncated histories stay valid. Evaluated on ChatGPT conversations and large models (OPT‑175B example), it reports TTFT reduced by up to 87%, up to 7.8× prompt prefilling, ~70% inference cost drop, and 9–16% storage overhead.
AI-written summary · about
chapters (7)
- 0:00Cost-efficient LLM serving
- 3:19Multi-turn inefficiency
- 5:54Reusing K caches in GPU memory
- 7:01Challenges of K cache caching
- 11:19Cache Attention technique
- 18:30Evaluating Cache Attention
- 21:04Conclusion
related talks
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
19mSystem for Scaling Large Language Models
17mDroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants
30mLarge Language Models and Prompt Engineering for Developers
25mEmbeddings, Transformers, RLHF: Three key ideas to understand ChatGPT
01h 00mInside GPT – Large Language Models Demystified
30mLLM Inference Arithmetics: the Theory behind Model Serving
26mEfficiently serving LLMs at scale.