Effortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 15:15· Pavilion 7 | Level 7.3 | Room S03·
summary
Orchestrates deployment of large open models like Falcon and Llama 2 on Kubernetes using Kubernetes CRDs and a built in HTTP inference server, packaging model weights inside container images to avoid repeated manual setup. Claims preset model images and automated inference tuning cut deployment from weeks to minutes, auto-provisions GPU nodes and can split inference across smaller GPUs (example Falcon 7B), reduces cost by sharing GPUs, currently supports AKS out of the box with roadmap toward on‑prem and cloud neutrality.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:10Kubernetes for ML deployments
- 5:59Kao the AI toolchain operator
- 8:30Kao automates LLM inference
- 11:45Live demo on AKS
- 17:24Deploying Falcon 7B
- 18:28Summary
- 19:15Q&A
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
28mProductionizing Machine Learning Models at Scale with Kubernetes
15mAI Model Inference on K8s
35mServing Large Language Models using KServe
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mDeploying Large Language Models with Ease
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…