Efficiently Deploying and Benchmarking LLMs in Kubernetes
summary
Deploys LLMs on OpenShift AI using the KServe stack and automates installs, CRDs and CI with topsell; a multiprocessing Python tool llm-load-test simulates concurrent users and reports tokens/sec, time-to-first-token, inter-token latency and error rate. Benchmarks include GPU DCGM metrics and Prometheus scrapes; a Llama 27B run on a G5 2xlarge (A10) shows ~$1.21 cost per million tokens at concurrency 10. Tests find weird throughput drops (Llama 23B) at concurrency 16 and intermittent variance across 12–24h longevity runs.
AI-written summary · about
chapters (15)
- 0:00Efficient LLM deployment
- 3:36LLM production complexities
- 4:43OpenShift AI for LLMs
- 6:53KServe LLM inference
- 7:57Performance test tooling
- 9:43Key performance indicators for LLMs
- 11:11Cost optimization with KPIs
- 13:21LLM load test config
- 15:27Automating with TopSell
- 18:15Test results visualization
- 20:46Model performance surprises
- 21:49GPU and memory analysis
- 24:00CI with TopSell and load test
- 27:30Future improvements
- 28:55Conclusion
related talks
40mScalable and Efficient LLM Serving With the VLLM Production Stack
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
27mYes You Can Run LLMs on Kubernetes
42mStop Treating LLMs Like REST APIs
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
32mScaling Generative AI Inference with llm-d
33mllm-d: Kubernetes Native Distributed Inferencing