LLM Quality Optimization Bootcamp
summary
Fine-tunes a Llama 3 8B model on 10,000 examples from the PImasking-200k dataset using OpenPipe, exports a 50MB LoRA and deploys it on OctoAI. Evaluation on 100 test samples shows fine-tuned model scoring ~0.97 vs GPT‑4’s ~0.68 (about 47% relative improvement), while serving cost drops from ~$30 to $0.15 per million tokens (~200x cheaper). Demonstrates Llama 3 8B 32K context, parameter‑efficient fine-tuning, function‑call PII redaction, and an end‑to‑end OpenPipe → OctoAI workflow.
AI-written summary · about
chapters (16)
- 0:00LLM fine-tuning challenges
- 7:24When to use fine-tuning for LLM quality
- 11:14Fine-tuning deployment cycle
- 12:18Building fine-tuning dataset
- 14:47Open vs closed-source models
- 17:37Parameter efficient vs full tuning
- 20:44DIY vs SAS for tuning
- 23:12Evaluating fine-tuned models
- 25:38Demo setup and use case
- 29:24Creating fine-tuning dataset
- 33:19Uploading to OpenPipe
- 35:03Fine-tuning with OpenPipe
- 37:31Deploying on Octo AI
- 46:13Evaluating accuracy and cost
- 49:52Key takeaways and next steps
- 52:41Q&A
related talks
09mTailoring Llama for Your Needs: Fine-Tuning & Evaluation
25mFinetuning Large Language Models (LLMs) for Security Log Detections
29mLocal LLM fine-tuning: a practical example
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
48mPracticalities of Fine-Tuning Llama 2 with AI Studio
03mFinetune Llama 4 Faster
27mProductionizing Open Source LLMs
36mBuilding & Deploying LLM Apps