Tailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM
summary
Compares two practical evaluation paths: benchmarking using task-specific datasets to catch training regressions and contamination, and LLM-as-a-judge using models (few-shot, chain-of-thought, ensembles) to score complex constraints like “gluten-free” or “10-minute no-oven” recipes. Describes a recipe model called Ratatouille, an automated pipeline at Intuit, and experiments showing human inter-rater agreement ~80–81% and human vs GPT-4 around 85%–90% with a fine-tuned judge.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker intro
- 1:13Personal cooking assistant demo
- 2:41Key evaluation criteria for LLMs
- 3:45Manual evaluation method
- 4:47Benchmarking method overview
- 6:55Creating custom benchmarks
- 8:04Running benchmarks with frameworks
- 9:29LLM-as-a-judge method
- 10:53Limitations of LLM-as-a-judge
- 11:57Reducing biases in LLM evaluation
- 12:40Reliability of LLM-based evaluation
- 13:27Summary and closing remarks
related talks
32mA Practical Guide to LLM Evaluation
35mEvaluating LLM systems: the nuts and bolts
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
28mTrustworthy LLMs: Vibe checks are not all you need
29mCustomizing and Evaluating LLMs, an Ops Perspective
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
38mMaking LLMs reliable - A practical framework
29mPutting the data science back into LLM evaluation