Overcoming Challenges in Serving Large Language Model
summary
Shows how to host GPT-style models on Kubernetes with multi-GPU nodes, demonstrating a live KServe demo of a 7B Bloom model and concrete trade-offs: a 6 GB model copy to pod storage takes ~90s and another ~25s to load to GPU, FP16 post‑training quantization roughly halves model size, and 175B-parameter GPT-3 maps to ~700 GB so multi-GPU GPU sharding or layer-wise placement is required. Covers model formats, registry-to-pod latency, warm‑up metrics, and using Huggingface Accelerate for tensor parallelism and FP16 quantization to cut memory.
AI-written summary · about
chapters (9)
- 0:00Serving LLM introduction
- 5:24Model parameters challenges
- 9:19Model serving basics
- 12:05Kubernetes for serving
- 16:47Prediction data importance
- 20:21Model size impact
- 24:16Handling large models
- 27:04Challenges and solutions
- 28:14Live demo with KServe
related talks
35mServing Large Language Models using KServe
28mTraining Large Language Models on Kubernetes
36mAccelerating High-Performance Machine Learning at Scale i...
21mOptimizing Model Serving on Kubernetes With Model Streaming
36mAccelerating Machine Learning at Scale with
40mServing Machine Learning Models at Scale Using KServing
37mAccelerate and Autoscale Deep Learning Inference on GPUs with KFServing
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine