Deploying Large Language Models with Ease
summary
Covers concrete self‑hosting choices: evaluates Mixtral, Llama3 and llama.cpp for domain tasks, prefers smaller models for lower latency and cost, and reports replacing large models to cut user response from 30–60s to latency down to 2–3s. Describes tooling choices (Titan ML, Triton, on‑prem Kubernetes), quantization with AWQ quantization that yields a 90% cost reduction on one model, continuous batching that gives 2x throughput, and hardware limits (A100/H100, 60s RPC cloud ceilings) and GPU supply risks.
AI-written summary · about
chapters (10)
- 0:00Introduction to self-hosting LLMs
- 3:33Challenges of commercial LLM APIs
- 7:02Model selection: Smaller is better
- 10:12Hosting framework struggles
- 13:42Hardware supply and demand issues
- 14:46Optimizing inference costs
- 18:22Batching vs streaming for efficiency
- 20:50User expectations for streaming
- 21:57Key takeaways and conclusions
- 22:45Q&A session
related talks
25mFine-Tuning Large Language Models With Declarative ML Orchestration
30mLearning to wield large language models
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
22mIntroduction to Large Language Models (LLM) on Kubernetes
43mLarge Language Models for Devs: from zero to your first LLM application
28mNavigating LLM deployment: Tips, tricks, and techniques
20mSelf-hosting AI LLMs - a beginners guide
33mTranslate tech talks using open-source models and cloud-native Wasm