Keynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
summary
A cloud-native LLM inference platform is built on Kubernetes and unifies open-source and vendor models behind an OpenAI-compatible API, using vLLM production stack for simple Helm-based deployment and NVIDIA Dynamo for distributed, aggregated or disaggregated serving. It implements KV-aware routing, prefix caching and NIXL GPU-to-GPU transfers, autoscaling-to-zero with KEDA (three triggers), fractional GPU packing via KAI-Scheduler, benchmarks with APerf reporting TTFT and latencies, and observability through Prometheus/Grafana/ELK.
AI-written summary · about
related talks
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
34mImprove AI Inference (serving models) With KServe and VLLM
33mScalable LLM Inference on Kubernetes With NVIDIA NIMS, LangChain, Milvus and Flu...
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d