PagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
summary
PagedAttention implements OS-style paging inside vLLM to split KV caches into 16–32 token pages, allocates pages on demand from a pool, tracks tokens with an indirection table, and reclaims pages after requests end. Integrated with vLLM on GPUs (A100 example), it reports up to 30× higher throughput, cuts KV cache waste to ~4%, halves latency (≈800ms→≈400ms) and drops cost per 1M inferences (~$12→$4.50), with lookup overhead and prefetching as active optimizations.
AI-written summary · about
chapters (13)
- 0:00PagedAttention and VLM intro
- 1:03Traditional LLM memory limits
- 3:53Memory fragmentation in LLMs
- 6:01What is PagedAttention and its key features
- 7:26Need for PagedAttention
- 8:32PagedAttention memory paging
- 11:01Dynamic memory allocation
- 13:54Page reuse and pool management
- 15:17PagedAttention benefits
- 16:24Real-world cost efficiency
- 18:54Integration and compatibility
- 19:39Future enhancements
- 21:04Q&A: Inference time
related talks
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
39mAccelerating Open-Source RL and Agentic Inference with vLLM
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
34mVLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
26mLMCache: Lower LLM Performance Costs in the Enterprise