FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
summary
FlexLLM co-serves LLM inference and PEFT finetuning on the same GPUs by fusing inference and adapter forward passes and splitting backward work across steps, using token-level finetuning and layer-level backward splitting plus a hybrid token scheduler to meet latency SLOs. A static compiler applies graph pruning and dependent parallelization to cut activation memory up to 80%, letting one cluster host LLaMA-3.1-8B and Qwen-2.5 models while sustaining SLOs at ~20 req/s and boosting finetuning throughput 1.9–4.8× (heavy) and 2.5–6.8× (light), preserving >76% peak progress; code is public at flexllm.github.io.
AI-written summary · about
chapters (14)
- 0:00FlexLLM introduction
- 1:42Current fine-tuning deployment
- 3:50Separate cluster challenges
- 4:53Co-serving concept
- 5:58FlexLLM architecture
- 7:20Memory optimization via pruning
- 8:21Parallelization challenges
- 10:49Naive multiplexing failure
- 11:54Incremental fine-tuning for SLO
- 13:21Evaluation setup
- 14:45SLO results comparison
- 16:09Alternative multiplexing
- 17:35Real-time burst adaptability
- 18:38Summary
related talks
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
29mLocal LLM fine-tuning: a practical example
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
19mNo GPU Left Behind: Scaling Online LLM Training With...
21mAssyLLM: Efficient Federated Fine-tuning of LLMs via Assembling Pre-trained Blocks
10mAuto-tuning vllm
35mThe State of vLLM
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models