SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs
summary
SolidAttention is an LLM inference engine that offloads the full KV cache to SSD by consolidating KV pairs into coarse-grained blocks and using speculative prefetching plus an SSD-aware scheduler to overlap I/O and computation. With a 128k-token context it reports up to 3.1× speedup and up to 98% KV cache memory reduction (16 GB context reduced to ~200 MB active VRAM) while keeping accuracy; on an RTX 4070/integrated GPU it shows <11% throughput loss versus an all-in-memory path.
AI-written summary · about
chapters (17)
- 0:00SolidAttention introduction
- 1:03LLM inference phases
- 2:53Quantization limits
- 3:38Attention sparsity and SSD
- 5:47SSD-based KV cache challenges
- 6:52Accuracy versus efficiency
- 9:00Data inconsistency risks
- 10:03SolidAttention architecture
- 11:05KV interleaving
- 12:07Prefetching KV cache
- 13:32Scheduler design
- 14:36Performance evaluation
- 15:45Throughput and memory results
- 16:29SSD overhead analysis
- 16:50Accuracy in long contexts
- 17:34Quantization failure
- 17:55Summary and future impact
related talks
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
14mSYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems
15mKV Cache: Accelerating AI Inference on Intel CPU
26mEfficiently serving LLMs at scale.
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
24mPortable High‑Performance LLM Serving: A Triton Backend for...
16mCognitive SSD: A Deep Learning Engine for In-Storage Data Retrieval
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...