Accelerating Model Loading in LLM Inference by Programmable Page Cache
summary
PPC implements a user-space, file-routing page cache that intercepts read events and runs per-model policies for prefetch and eviction; MAIO uses an I/O-template engine with XPU/NUMA affinity and preemptable grouped reads to saturate SSD bandwidth and avoid cache thrash. Evaluation shows MAIO reduces model loading latency by up to 79% and yields up to 36% higher inference throughput in elastic deployments with only ~3–4% subsystem overhead, plus a new burn after reading eviction mode.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:01LLM inference on edge devices
- 3:05Existing optimization limits
- 5:12File system cache inefficiencies
- 7:15Programmable page cache design
- 9:27How programmable cache works
- 11:15Page cache subsystem operations
- 12:42Model-accelerated IO system
- 14:28Burn after reading eviction policy
- 15:09Evaluation setup
- 16:37Performance under memory constraints
- 18:03Performance gains breakdown
- 18:46Real-world deployment results
- 19:29Key findings
related talks
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
24mCost and Performance Optimization of LLM Inferencing.
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
02mInferentia2
34mOptimizing AI Inference with ML Compilers & Hardware
39mAccelerating Open-Source RL and Agentic Inference with vLLM
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
06m⚡ Eliminating Adaptation Latency: Day-0 Inference for New Models on Multi-Backends With...