KV Cache: Accelerating AI Inference on Intel CPU
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Describes KVShrink, a system that offloads LLM KV caches from GPU HBM into DDR or storage and compresses them using Intel QAT hardware, treating the KV cache as a tiered, compressible asset. Shows how keeping more cache entries available across requests raises KV hit rates so GPUs avoid redundant recomputation, reducing time-to-first-token latency and enabling higher concurrency without buying additional GPUs.
AI-written summary · about
related talks
16mNo Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
15mECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native...
33mScaling KV Caches for LLMs: How LMCache + NIXL Handle Network and Storage...
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
44mVLLM K/V Caching With Ceph