DevOps for Private AI: A Scalable Platform Using Kubeflow
summary
A platform for private AI built on Kubernetes and Kubeflow, demonstrating multi-GPU training pipelines and concrete Kubernetes manifests for fine-tuning. It covers production model deployment using KServe with VLLM serving, canary releases, observability metrics, and compares inference engines (VLLM, Triton, TGI), finding VLLM most performant. It also presents an operator-based 0→n autoscaling solution and in-cluster routing to serve multiple models from a single endpoint, plus note that Hugging Face hosts hundreds of thousands of models.
AI-written summary · about
chapters (11)
- 0:00DevOps for private AI
- 1:07Private AI and LLMs
- 2:35Data sovereignty benefits
- 4:46Hosting LLMs privately
- 6:11Hugging Face open-source models
- 7:15Kubeflow for AI lifecycle
- 9:02KServe scalable inference
- 10:28LLM inference engines compared
- 12:59QBI for LLM inference
- 14:00AI agents 2025 trends
- 16:30Q&A
related talks
46mDeploying scalable Machine Learning models with Kubernetes
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
44mEngaging the Kubeflow Community: Building an Enterprise-Ready AI/ML Platform
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
17mChallenges and takeaways of managing AI workloads on cloud
51mMachine Learning using Kubeflow and Kubernetes
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
29mA Scalable Platform for Training and Inference Using Kubeflow