InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Framework that offloads the KV cache to CPU memory and speculative KV prefetching reduces PCIe transfers by predicting and fetching only the small set of tokens needed for the next attention layer, overlapping transfer with prior-layer computation. Prediction runs a minimal rehearsal using part of the query weight and a cached key subset plus an offline SVD key-query sking to preserve exact dot products. Evaluation on Opt and LLaMA-2 against CUDA UVM, FlexGen and H2 shows up to a 3.00× speedup while maintaining model accuracy.
AI-written summary · about
chapters (11)
- 0:00InfiniGen intro
- 1:07Long-context LM inference challenges
- 2:53Prior KV cache limits
- 5:44Why prior methods fail
- 6:06CPU offloading in InfiniGen
- 8:11Essential token prediction
- 11:01Key Query Skipping
- 13:08Evaluation setup & gains
- 14:13Accuracy & scalability
- 15:18Conclusion & future
- 15:59Q&A
related talks
41mSub-Linear Algorithms meets Large Language Models
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
30mLLM Inference Arithmetics: the Theory behind Model Serving
12mKV Cache and Long-Context Inference: Solutions for Inference at Scale
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
24mKServe Next: Advancing Generative AI Model Serving
38mOptimizing LLM Inference for the Rest of Us
34mKServe in Production: Scaling Generative Inference the Cloud Native Way