Orchestrating AI Models in Kubernetes: Deploying Ollama as a Nati...
summary
Shows a working pipeline that runs Ollama models natively on Kubernetes by making models OCI images and adding a custom containerd runtime. Uses GGUF packer go to convert GGUF models (e.g., Llama 3 8B / Q12) into OCI images, copies a small containerd shim binary onto nodes, registers a runtime class, mutates the container bundle to mount a model file and invoke Ollama, and demos it on kind plus integration into a QFlow model registry and UI, end-to-end and functional.
AI-written summary · about
chapters (10)
- 0:00AI model orchestration in Kubernetes
- 3:11What is Ollama and Kubernetes
- 9:20Kubernetes custom runtimes
- 13:25AI models to OCI images
- 15:43Custom containerd runtime for Ollama
- 19:36Ollama models with Kubeflow
- 23:08Model catalog UI in Kubeflow
- 26:40Deploying Ollama standalone
- 29:28Ollama with Kubeflow platform
- 31:15Summary and future
related talks
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
24mDemo of Kubernetes at the Edge for AI Model Deployments
44mFrom Containers To Cognitive Agents: Open Source Foundations for Enterprise-Grad...
50mRunning Open LLM Models on Kubernetes
43mRunning open large language models in production with Ollama and serverless GPUs
22mBuilding a MLOps Platform with Kubeflow on GKE
25mFine-Tuning Large Language Models With Declarative ML Orchestration