Vertical Mobility: Inference from MVP to Trillion Parameters
summary
A multi‑tier inference platform is shown that scales from pay‑per‑token experiments to single‑tenant GPU leases and self‑managed clusters, implemented as serverless vs dedicated vs CKS. It uses vLLM, S L ang and TensorRT‑LLM engines, applies four levers—caching, disaggregation, quantization and speculative decoding—and reports 95%+ cache reuse in agentic workloads and 70–75% reuse for chat. Infrastructure supports cache offload (LM Cache/Moon Cake), per‑GPU billing on dedicated nodes, and claims top throughput on both synthetic and real traffic (Kimiko 6/27, MiniMax M3, GLM 52).
AI-written summary · about
chapters (7)
- 0:00Introduction to vertical mobility
- 1:08Workload profiles overview
- 3:59Product offerings explained
- 5:45Platform workflow and routing
- 8:15Performance optimization levers
- 10:45Performance benchmarks and results
- 11:28Q&A
related talks
31mThe Evolution of Multi-GPU Inference in vLLM
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
21mScaling LLM Inference with vLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
24mLLM Inference Optimization From Simple To Sophisticated
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
28mShared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…