llm-d: Kubernetes Native Distributed Inferencing
summary
llm-d serves LLMs by marrying VLM-style inference with Kubernetes: it implements continuous batching to pack per-step token work, manages massive KV cache page retention like virtual memory (roughly 1 GB per 10k tokens for Llama-370B), and wires a Gateway API inferenceuler for prefix-cache-aware routing so repeated prompts hit the same pod. It supports KV transfers (GPU‑direct/RDMA via Nixel) and a prefill/decode disaggregation that shows ~25% user throughput gain on an 8‑GPU 4k in/4k out benchmark.
AI-written summary · about
chapters (15)
- 0:00LLM inference at Red Hat
- 1:03What is VLM
- 2:50Specialized LLM inference software
- 5:57Auto-regressive model challenges
- 8:04Dynamic vs continuous batching
- 10:12KV cache impact on performance
- 13:27Page attention and virtual memory
- 17:18How page attention manages memory
- 19:01Prefix caching benefits
- 20:07LLM vs standard HTTP requests
- 23:21LMD architecture and Kubernetes
- 24:48Prefix cache-aware routing
- 26:34Prefill-decode disaggregation
- 29:43Offloading KV cache
- 31:12Summary and future of LLM systems
related talks
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
32mScaling Generative AI Inference with llm-d
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
38mOptimizing LLM Inference for the Rest of Us
42mStop Treating LLMs Like REST APIs
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…