The Token Slice: Implementing Preemptive Scheduling Via Chunked Decod...
summary
An experimental Chunked Decoding system slices generation into fixed-size token time slices and runs as a sidecar implementation for PyTorch servers (example: vLLM) to enable preemptive scheduling without discarding KV caches. Implemented inside the LMD inference scheduler (Kubernetes-first), it interleaves requests midstream, targets multi-tenant fairness, and evaluates chunk-size effects on priority handling and tail latency. Trade-offs include context-switching and potential KV recompute; tests reference a scenario with 16 H100 GPUs serving eight Qwen 32B models at ~70% KV utilization.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:44What is LLMD
- 3:09Well-lit paths in LLMD
- 5:17Prefill decode disaggregation
- 6:00Intelligent inference scheduling
- 9:12KV cache management
- 10:36Token slicing concept
- 12:19Token slicing implementation
- 13:23Token slicing challenges
- 14:49Flow control and preemption
- 16:34Eviction vs preemption
- 17:58Future work
- 18:19Q&A
related talks
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
28mMultitenancy and Fairness at Scale with Kueue: A Case Study
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM
30mTopology-Aware Scheduling for AI Training & Inference With Kueue
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
32mDynamic Scheduling for Large Language Model Serving
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
34mDeep Customized Kubernetes for Machine Learning in Tencent