SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
summary
SuperServe implements SubNetAct and SlackFit to serve many weight‑shared sub-net models from a single pre-trained super-network by inserting control‑flow operators that SubNetAct actuates sub-nets in <1 ms, enabling per-request routing and dynamic batching. The system jointly picks model and batch size using an EDF slack heuristic (fine‑grained reactive scheduling) and runs on real Microsoft traces, yielding up to 2.6× lower memory, 4.67% higher mean accuracy and 2.85× higher latency‑target attainment versus prior baselines.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:54Bursty workload challenges
- 4:43Prior systems limitations
- 8:14Model switching latency
- 10:43SlackFit policy
- 12:47Experimental evaluation
- 14:35Real-world burst handling
related talks
Power-aware Deep Learning Model Serving with μ-Serve
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
12mPreview: ML Serving
40mScalable machine-learned model serving
17mModel-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service
31mServing Machine Learning Models at Scale Using KServe
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)