Running Open LLM Models on Kubernetes
summary
A practical introduction and demo showing what’s needed to run open-source LLMs (transformer-based models) in your own Kubernetes cluster. Covers data collection and tokenization, model architecture, training and optional fine-tuning, evaluation, saving and deployment, plus discussion of Hugging Face, managed cloud services, Kubernetes deployment patterns, and LLM serving frameworks with a live demo.
AI-written summary · about
chapters (19)
- 0:00Intro
- 3:23What is LLM?
- 5:03What is Transformers?
- 5:35Data Collection and Preparation
- 6:54Tokenization and Vocabulary Building
- 8:00Model Architecture Design
- 9:33Training
- 13:09Fine-Tuning (Optional)
- 15:25Evaluation and Testing
- 16:06Saving and Deployment
- 16:58TL;DR
- 17:51Hugging Face - GitHub for LLMs
- 19:44How to run?
- 22:27Using Managed Cloud Services
- 23:11Kubernetes
- 25:50LLM Serving Frameworks
- 30:14Demo Time
- 40:32Conclusions
- 41:59Q&A
related talks
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
22mIntroduction to Large Language Models (LLM) on Kubernetes
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer
30mLearning to wield large language models
21mStreamline LLM Fine-Tuning on Kubernetes with Kubeflow LLM Trainer
10mLightning Talk: LLM-Driven Applications: Intro to LLM Agents in Kubernetes
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes