HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
summary
HydraServe implements pipeline parallelism, stage overlapping, and pipeline consolidation to cut serverless LLM cold starts that can reach 45 seconds (model fetch ~25s, GPU init ~10s). It shards model layers across workers to aggregate network bandwidth, overlaps container/CUDA/library setup with model fetch and host→GPU streaming via a prefetcher and parameter manager, and later merges shards back to single-node inference. Evaluations show 1.7×–4.7× TTFT reduction, 1.43×–1.74× SLO attainment improvement, modest TPOT impact, and overall cost savings.
AI-written summary · about
chapters (8)
- 0:00Introduction to HydraServe and serverless LLM serving
- 2:49Cold start problem
- 5:39Pipeline parallelism
- 7:46Stage overlapping
- 9:11HydraServe architecture
- 13:20SLO-aware allocation
- 14:44Evaluation results
- 16:30Summary
related talks
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
27mServerless Performance on a Budget
42mStop Treating LLMs Like REST APIs
47mAdvanced Serverless Architectural Patterns on AWS
26mEfficiently serving LLMs at scale.
24mCost and Performance Optimization of LLM Inferencing.
18mServerless AI's Rise: Revolutionizing ML Deployment
24mChallenges with Ultra-low Latency LLM Inference at Scale