Cloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
summary
vLLM runs as a Kubernetes-native inference server that maximizes throughput by combining quantized models, KV cache/prefix caching, and dynamic batching; a demo shows a Blackjack endpoint handling about 1.2k requests during an event and a local vibe‑coding IDE using a private endpoint. Models split across nodes and can be sharded across 4 GPUs for multi‑GPU or multi‑node inference; tiny models run on CPU, and one live demo fails but a backup runs successfully.
AI-written summary · about
chapters (14)
- 0:00Talk overview
- 1:05vLLM over other runtimes
- 3:11Local AI models
- 8:25Local to Kubernetes
- 9:30What makes vLLM unique for Kubernetes
- 12:22vLLM for inference challenges
- 15:11Efficient vLLM deployment
- 17:01Quantization and hardware
- 19:55Distributed GPU inference
- 22:08Live demo: Kubernetes serving
- 24:40Vibe coding with vLLM
- 27:07Blackjack with Mistral
- 30:37LLMD for distributed inference
- 31:19Q&A
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
34mImprove AI Inference (serving models) With KServe and VLLM
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM
32mScaling Generative AI Inference with llm-d
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d