Production-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 15:15· Level 1 | Hall Entrance S10 | Room A·
summary
Deploys running on 1–2 A100 GPUs using Helm, Open Web UI, Ollama for quick tests and vLLM for production-grade inference with Mistral 7B; finds 4-bit quantization garbles answers so moves to 8-bit, enables Flash Attention to cut context-memory from quadratic to subquadratic. Weights are baked into Docker images (had to replace gzip compression), built a custom scheduler for GPU/memory sharing, added an eval CLI that grades outputs, and a vision RAG pipeline that reads table images (returned 0.35 correctly); defines an AI-spec CRD to declare RAG/API integrations as config.
AI-written summary · about
chapters (15)
- 0:00Running LLMs on infrastructure
- 3:13What is an LLM and why it matters
- 4:19LLMs on Kubernetes
- 5:24Self-hosting LLMs securely
- 7:47Demo: Ollama with Open Web UI
- 10:41Ollama pitfalls in production
- 12:49Switching to vLLM
- 15:02Model weights in air-gapped setups
- 16:27Serving multiple models on K8s
- 17:51LLM layers for business value
- 18:33LLMs with APIs and knowledge
- 21:44Retrieval augmented generation with vision capabilities
- 23:32Demo: Advanced vision RAG
- 27:04Kubernetes-native GenAI spec
- 27:47Q&A
related talks
38mOptimizing LLM Inference for the Rest of Us
42mStop Treating LLMs Like REST APIs
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
29mLeveraging open-source LLMs for production
23mLLMs on the Edge: The Future of On-Device Intelligence
40mScalable and Efficient LLM Serving With the VLLM Production Stack
33mScalable LLM Inference on Kubernetes With NVIDIA NIMS, LangChain, Milvus and Flu...