ServeGen: Workload Characterization and Generation of Large Language...
summary
Characterization uses production traces spanning 3.5 billion requests over 4 months across 10 models and finds that traffic is dominated by a few customers — 1% clients produce 90% of traffic — each with stable but distinct arrival, input/output length, and modality mixes. Multimodal requests show long tails (up to 63 audio items), reasoning sessions average 3.1 turns with ~100s think time, and naive Poisson+sample generation underestimates needed capacity, so ServeGen composes workloads per client and is open-sourced.
AI-written summary · about
chapters (12)
- 0:00ServeGen introduction and goals
- 1:23Limitations of prior LLM workloads
- 3:07ServeGen’s three contributions
- 3:50Complex arrival patterns in LLM
- 5:15Request length distributions
- 7:45Client decomposition patterns
- 9:54Multimodal and reasoning workloads
- 12:26ServeGen’s generation framework
- 13:30Case study: provisioning accuracy
- 14:11Case study: disaggregation sensitivity
- 14:53Future directions and release
- 15:33Q&A
related talks
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
17mCharacterization of Large Language Model Development in the Datacenter
37mLoad testing speech synthesis and recognition at SberDevices
11mStreaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving
44mIntroduction to LLM serving with SGLang
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
33mProfiling Code
21mMulti Runtime Serving Pipelines