Productionizing Open Source LLMs
summary
Open source LLMs now approach GPT‑4 on some tasks (Llama 70B ~81.7% vs GPT‑4 85.5% on a summarization study) and fine‑tuned smaller models (7B/13B) can beat GPT‑4 on function call and SQL generation. A production recipe is concrete: use OpenLLM with BentoML packaging, quantize 70B to 3–4 bits to fit under ~40GB on one A100, enable continuous batching, model parallelism when needed, and token streaming; these optimizations claim workable latency, throughput, and cost savings.
AI-written summary · about
chapters (17)
- 0:00Open source LLMs intro
- 2:39Why productionize LLMs
- 4:05Control and fine-tuning
- 7:24Data privacy in production
- 8:27Cost efficiency of LLMs
- 10:13Operability challenges
- 11:20Scalability and cost
- 13:09Throughput and batching
- 14:34Latency types and fixes
- 15:39AI apps with LLMs
- 18:07OpenLLM for production
- 20:13OpenLLM key features
- 21:17Quantization and optimization
- 22:02Batching and parallelism
- 23:46Token streaming
- 24:30MLOps with BentoML
- 26:37Conclusion
related talks
01h 02mA hacker's guide to open source LLMs
37mExpanding the Capabilities of LLMs Through Community Collaboration
24mHarnessing the Power of LLMs in Advanced Analytics
19mSystem for Scaling Large Language Models
29mRunning Large Language Models on Laptops...
22mThe Disruptive Potential of On-Device LLMs
20mLLM: Efficiency and Sustainability in Practice
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs