Stop Treating LLMs Like REST APIs
summary
Claims that treating LLM inference like REST causes launch-day failure, and shows a concrete fix: deploy LLMD on Kubernetes with disaggregated prefill vs decode across a hybrid NVIDIA/AMD node pool, use session affinity and caching, and a ready GitHub quickstart that spins a 4-node cluster (2 CPU + 2 GPU) and LLMD stack in ~15 minutes. Real numbers: user load from 1k to 100k inflates cost from $5k to $150k and latency from ~200 ms to ~8 s; the demo completes with Grafana dashboards showing token throughput and lower tail latency.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:13Inference workloads dominate AI
- 8:31REST API problems for LLMs
- 12:26Introducing LLMD
- 13:08LLM inference crash course
- 16:18LLMD’s disaggregated serving
- 19:13Live demo: LLMD scheduling
- 26:22Deploy LLMD on Kubernetes
- 39:03Deployment recap and next steps
- 41:31Resources and closing
related talks
38mOptimizing LLM Inference for the Rest of Us
28mChildren's Guide To LLMs on Kubernetes
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
33mllm-d: Kubernetes Native Distributed Inferencing
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
10mLightning Talk: Best Practices for LLM Serving with DRA