Customizing and Evaluating LLMs, an Ops Perspective
summary
Clear claim: customizing and evaluating LLMs is messy but solvable with concrete steps. Practical wins include a fraud‑detection gullible agent deployed in production that captured identifiers, retrieval augmentation via RAG to keep context current, parameter-efficient domain shifts using Laura, and experiment tracking with mlflow plus Hugging Face and Label Studio for human preference tests. Emphasis is on measurable metrics, domain‑specific test suites, automated plus human evaluation, and whether each pipeline component actually reduces error in live use.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:06LLM customization necessity
- 3:37LLM customization beyond chatbots
- 5:43Domain-specific customization
- 7:53Obstacles in LLM customization
- 9:40LLM customization strategies
- 11:25Prompt engineering techniques
- 13:32Retrieval Augmented Generation
- 14:57LoRA fine-tuning
- 16:22Fine-tuning deep dive
- 17:04Reinforcement learning from feedback
- 18:48Combining customization approaches
- 19:11LLM evaluation challenges
- 20:58Automated evaluation methods
- 21:59Specialized evaluation datasets
- 22:45Meta evaluation with LLMs
- 23:27Production feedback validation
- 24:32Customization tools overview
- 25:55Key takeaways
- 26:20Q&A
related talks
35mEvaluating LLM systems: the nuts and bolts
38mMaking LLMs reliable - A practical framework
28mTrustworthy LLMs: Vibe checks are not all you need
32mA Practical Guide to LLM Evaluation
29mLocal LLM fine-tuning: a practical example
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications