Scalable LLM Inference on Kubernetes With NVIDIA NIMS, LangChain, Milvus and Flu...
summary
Blueprint for a retrieval‑augmented chatbot running on Amazon EKS that actually deploys embedder and LLM as NVIDIA NIMS, stores vectors in Milvus (uses GPU‑optimized indices and S3 persistence), and uses FluxCD for GitOps plus Karpenter to spin G5 nodes on demand. Uses llama 1B quantized profiles (TensorRT/FP16/FP8), NIM cache on EFS to avoid repeated pulls, Prometheus+adapter→HPA for pod autoscaling based on custom metrics, and a published GitHub repo with working manifests.
AI-written summary · about
chapters (8)
- 0:00Scalable RAG chatbots on Kubernetes
- 2:58Retrieval augmented generation
- 7:58Kubernetes cluster setup
- 13:51NVIDIA NIMs for LLM deployment
- 18:29Milvus GPU vector search
- 23:00Scaling Milvus with Prometheus
- 26:35Flux GitOps deployment
- 28:01Q&A on GPU usage
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
32mScaling Generative AI Inference with llm-d
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
33mllm-d: Kubernetes Native Distributed Inferencing
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
22mIntroduction to Large Language Models (LLM) on Kubernetes
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes