Introduction to Large Language Models (LLM) on Kubernetes
summary
Beginner walkthrough of running a pre-trained chat LLM container on Kubernetes using the AMA runtime, showing a live demo that successfully serves chat queries. Covers model sizes (8B, 70B, 405B) with rough VRAM needs (~5GB, ~40GB, ~200GB), GPU access via device plugins and the nvidia.com/gpu resource request, node taints and read-only model mounts, and the tradeoffs of utilization versus cost. Notes sequential inference on a single GPU and a cold start ~30s for model load.
AI-written summary · about
chapters (7)
- 0:00Introduction to LLMs on Kubernetes
- 1:47What is a large language model
- 5:24LLM use cases and limitations
- 9:59Why run LLMs on Kubernetes
- 14:14Kubernetes device plugins for GPUs
- 16:43Deploying an LLM on Kubernetes
- 20:46Q&A
related talks
27mYes You Can Run LLMs on Kubernetes
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
38mOptimizing LLM Inference for the Rest of Us
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
31mDeploying Large Language Models with Ease
30mLearning to wield large language models
50mRunning Open LLM Models on Kubernetes