Running and Fine-tuning Open Source LLMs on Google Kubernetes Engine
summary
Deploys a Llama 2 70B model on Google Kubernetes Engine using Nvidia L4 GPUs, mounting model and checkpoints to Cloud Storage with GCSFuse, and serves via TGI while comparing to vLLM for throughput and memory. Fine-tuning uses parameter-efficient methods like LoRA/adapters to avoid full-weight training, orchestrated as Kubernetes Jobs with node autoprovisioning to spin up extra GPU node pools. Demo shows a live TGI endpoint returning 20 tokens; checkpoints and Hugging Face-sourced data live in a GCS bucket.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 0:34Agenda
- 1:37Open Source LMS
- 2:26Why Open Source LMS
- 4:35Why Kubernetes
- 5:48What is Kubernetes Engine
- 8:12Node Autoprovisioning
- 9:45GPU Sharing
- 12:03Frameworks
- 14:01YAML File
- 15:22Customization
- 17:34Storage Use
- 19:16Job Queing
- 20:28Architecture
- 23:05Finetuning Job
- 23:37Finetuning Key Considerations
related talks
27mYes You Can Run LLMs on Kubernetes
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
12mUnleashing Kubernetes Intelligence - running k8sgpt utilizing your own fine-tuned LLM
28mTraining Large Language Models on Kubernetes
50mRunning Open LLM Models on Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer
22mIntroduction to Large Language Models (LLM) on Kubernetes