DEEPSERVE: Serverless Large Language Model Serving at Scale
summary
DEEPSERVE is a serverless LLM platform that serves models on Huawei Ascend NPUs with a concrete request-job-task model abstraction, an in-house FLOWSERVE engine using a microkernel design and SPMD NPU-centric execution, and scheduling that chooses between PD-disaggregated and PD-colocated placements. It runs on a 48-node super pod (384 NPUs), uses pre-warmed pods, DRAM pre-loading and NPU-fork to cut cold starts, scales up to 64 instances in seconds, has been in production over a year, and offers fine-tuning, agent, and model serving APIs.
AI-written summary · about
chapters (6)
- 0:00DeepServe introduction
- 3:27Overall architecture
- 8:22FlowServe engine design
- 13:27Request scheduling challenges
- 14:11Fast scaling strategies
- 15:37Recent progress
related talks
10mvLLM & Deepspeed Updates
26mEfficiently serving LLMs at scale.
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
17mHydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
35mServing Large Language Models using KServe
40mServing Machine Learning Models at Scale Using KServing
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference