Transform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
Lightningat Cloud Native + Kubernetes AI Day NA 2024 · on Tuesday, 2024-11-12· at 17:00· Salt Palace | Level 1 | Grand Ballroom A·
summary
LLMariner turns a Kubernetes cluster into a GenAI platform by providing an OpenAI-compatible API for inference, embeddings, fine-tuning and RAG while integrating with vLLM and NVIDIA Triton for serving and Q for DPU quota. It demonstrates a working VS Code Continue coding assistant using NVIDIA Llama 3.1 and a CLI that issues API keys. Multi-cluster support uses control-plane forwarding with worker-to-control persistent connections and a database-backed job tracking model instead of CRDs to simplify notebook and fine-tune lifecycle.
AI-written summary · about
related talks
09mKeynote: Kubernetes for GenAI: The New Operating System of the AI-First World
32mKeynote: Powering Generative AI with Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
15mKeynote: From Platforms to AI Factories. Has Kubernetes Solved It?
43mKeynote: Building AI in the Open: How the Cloud Native Community Shapes What Comes Next
38mAI in Kubernetes: How to Get Started?
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme
30mDesigning a Hybrid AI Platform on Kubernetes