VLLM K/V Caching With Ceph
summary
Prototype Ceph K/V caching plugin for vLLM offloads attention states to clustered NVMe and complements GPU HBM, using LM Cache with 256-token content-addressable blocks (≈62 MB / 256 tokens for a qwen‑32B example). A 2U four‑node Ceph RGW cluster with six NVMe per node and a 400G fabric sustains ~60 GB/s reads from a single GPU host. Experiments with 100k–130k contexts show 5x–10x reduction in time‑to‑first‑token, demonstrating practical latency and cost tradeoffs.
AI-written summary · about
chapters (8)
- 0:00Introduction and speakers
- 1:05What is KV caching
- 8:36Offloading KV data to storage
- 18:31Experiment setup with VLLM
- 26:46Read path and cache benefits
- 28:49Performance results and speedups
- 30:34Shared storage benefits and future work
- 33:50Q&A session
related talks
33mScaling KV Caches for LLMs: How LMCache + NIXL Handle Network and Storage...
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
16mNo Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs
34mVLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
15mKV Cache: Accelerating AI Inference on Intel CPU
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe