Yes You Can Run LLMs on Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 12:00· Level 1 | Hall Entrance N10 | Room F·
summary
Deploys a Gemma 3 1B model as a Kubernetes Deployment using vLLM image on GKE Autopilot, requesting 4 CPU, ~20GB RAM, 1 GPU and 20GB ephemeral storage; model download and server startup takes about four minutes. Demonstrates serving via Services, mixes TGI and Ollama images for other models, runs Llama 3 on TPU via vLLM, and shards DeepSeek R1 across two nodes (16 H100 GPUs total) using Ray plus a Leader Worker Set pattern, with PVC/HDML caching to avoid repeated 800GB downloads.
AI-written summary · about
chapters (15)
- 0:00Running LLMs on Kubernetes
- 1:05LLMs deployment challenges
- 3:31Kubernetes for LLMs
- 3:52Live demo: GKE LLM deployment
- 10:14LLM concepts explained
- 14:23Model serving engines
- 17:15Single vs multi-GPU deployments
- 18:17Optimizing LLM workloads
- 19:22Demo deployment status
- 20:26Interactive multi-model demo
- 22:14Serving engines in demo
- 23:15DeepSeek R1 deployment
- 24:39Open source resources
- 25:21Kubernetes AI reference architecture
- 26:44Q&A
related talks
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
01h 15mWorkshop: Working with Gemma and Open LLMs on Google Kubernetes...
43mRunning open large language models in production with Ollama and serverless GPUs
22mIntroduction to Large Language Models (LLM) on Kubernetes
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
40mScalable and Efficient LLM Serving With the VLLM Production Stack
05mBuild your own LLM on Cloud
12mDemo: Building cloud-native, AI-powered applications with GKE