ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
ServerlessLLM is a distributed system that actually stores checkpoints on GPU servers, implements a loading-optimized checkpoint format and a multi‑tier pipelined loader to saturate SSD→host→GPU bandwidth, and reports 5–8× faster load times. It implements live migration via token transfer and KV-cache recomputation to move inferences with millisecond interruptions, adds locality‑aware scheduling that chooses migration vs. direct load, and shows 10–200× lower latency, 4× fewer GPUs for equal latency, and >5× higher throughput in evaluations.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:48Cold starts in serverless
- 3:11Model loading dominates latency
- 5:02Checkpoint format optimization
- 7:09Fast loading system design
- 9:01Live migration importance
- 11:11Efficient live migration
- 12:39Fast loading + live migration
- 14:55Performance results
related talks
15mResource Multiplexing in Tuning and Serving Large Language Models
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
24mLLM Inference Optimization From Simple To Sophisticated
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
42mStop Treating LLMs Like REST APIs
24mChallenges with Ultra-low Latency LLM Inference at Scale
24mCost and Performance Optimization of LLM Inferencing.