Leveraging open-source LLMs for production
summary
A practical survey of what actually works: LLaMA 2 7B/13B/70B and Mistral 7B appear as usable bases, Code Llama for code, and Hugging Face benchmarks show a 70B LLaMA needs multi‑A100 rigs with ~41 ms per token and ~$4,200 per million tokens raw. Combining quantization (int8/int4) and LoRA adapters cuts memory and cost massively — examples show fine‑tuned 7B models matching much larger models on SQL and code tasks — and reinforcement learning or DPO improves alignment.
AI-written summary · about
chapters (9)
- 0:00Open-source LLMs in production
- 1:47Open-source vs proprietary models
- 4:54Key open-source LLM models
- 13:22Cost and performance challenges
- 16:12Quantization for efficient inference
- 19:29Low-rank adaptation for fine-tuning
- 23:10Fine-tuning and reinforcement learning
- 27:00Advanced techniques and future trends
- 28:29Open-source LLM deployment tools
related talks
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
47mOwn Your LLM: Running Generative and Agentic AI Locally
35mOpen-source AI: why it matters and how to get started
40mOpen Source LLMs in the Cloud: Scalable Solutions
35mLocal LLMs on low end hardware - a practical perspective
20mSelf-hosting AI LLMs - a beginners guide