KV Cache and Long-Context Inference: Solutions for Inference at Scale
summary
Offloading KV cache to VAST Data’s flash layer with NVIDIA Dynamo and BlueField‑4/CMX shifts recompute to reads, tested on H100/H8 hardware with 128k context over dual 100GbE links. Benchmarks claim 20x time to first token, 90% GPU time savings and up to 64% lower three‑year TCO. A 1.2 TB KV cache sample yields ~1.4:1 data reduction. The system exposes S3/RDMA and a BlueField DPU local‑SSD tier to build a shared pod‑level KV cache.
AI-written summary · about
chapters (10)
- 0:00Introduction and overview
- 0:28KV cache in long-context inference
- 1:32Cost and performance challenges
- 2:34Vast Data integration in Nvidia stack
- 3:58Optimizing KV cache management
- 5:23Experimental performance results
- 6:49Cost savings and data reduction
- 8:14Security and compliance for KV cache
- 9:17Future enhancements with CMX and DPUs
- 10:21Key benefits and summary
related talks
17mMooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture...
26mLMCache: Lower LLM Performance Costs in the Enterprise
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
26mEfficiently serving LLMs at scale.
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
20mHacking the Inference Pareto Frontier