Resource Multiplexing in Tuning and Serving Large Language Models
summary
LLMStation is a runtime that co-schedules parameter-efficient fine-tuning and inference on the same GPU by combining iteration-level multitasking scheduling, a suspendable Autograd that turns backward into resumable coroutines, and an inference/fusion engine that shares base model weights to cut memory movement. It claims 1.38×–14.77× throughput versus prior systems while meeting 50–100 ms inference SLOs, reports up to ~40× gains in low-load LLaMA experiments on a single GPU, and shows low suspend/resume overheads on single-GPU and modest overheads on multi-GPU.
AI-written summary · about
chapters (13)
- 0:00Resource multiplexing in LLMs
- 1:42LLM serving systems today
- 2:48LLM inference and fine-tuning challenges
- 4:33Sharing resources between tuning and serving
- 5:18Why temporal multiplexing fails
- 6:47Limitations of spatial multiplexing
- 7:30LMStation for efficient co-execution
- 8:55LMStation design principles
- 9:39Iteration-level multitasking scheduler
- 11:06Suspendable autograd engine
- 12:11Inference and fusion engine
- 12:54LMStation vs baselines
- 14:20Overhead analysis and summary
related talks
16mLlumnix: Dynamic Scheduling for Large Language Model Serving
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
14mScalable Solutions for Running Large Language Models
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
26mEfficiently serving LLMs at scale.
34mBeyond Static Pods: Dynamic GPU Sharing and Low-Latency Model Switching for LLM ...
32mDynamic Scheduling for Large Language Model Serving