Workshop: Optimizing Kubernetes Cluster Scaling for Adv...
summary
Shows a runnable path: Terraform + Civo spins a GPU-backed Kubernetes cluster in ~10 minutes and deploys Olama + OpenWebUI hosting Llama‑3 (demo succeeds though live demos sometimes lag). Uses Docker images, Helm charts and a $250 free Civo credit for hands‑on. Recommends vLLM dynamic batching and paged attention to raise throughput vs. one‑request/sec on a bare A100, and uses MIG / dynamic MIG (Kubernetes DR support from 1.26, enable in 1.30) to shard GPUs and improve utilization.
AI-written summary · about
chapters (9)
- 0:00AI model deployment intro
- 7:27Kubernetes for AI workloads
- 12:30Deploy Llama 3 with Terraform
- 30:37GPU utilization challenges
- 37:04Optimizing GPU with vLLM
- 47:52Dynamic GPU allocation with DR
- 1:00:02Demo: MIG with NVIDIA A100
- 1:08:05Alternative orchestrators
- 1:16:10Q&A
related talks
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
02h 26mKubernetes Performance Tuning Workshop
33mFrom spreadsheet scheduling to Kubernetes: building an on-premise ML platform
16mHow We Built a Cloud On K3s: The Learnings Of Growing Up Fast
27mHarnessing Kubernetes for Scalable AI/ML Workloads
38mBeyond Kubernetes: Adapting To Spe...
11mCost and Resource Optimization In Kubernetes
28mGPU sharing done right: Secrets, security, and scaling with Vault and vCluster