Judge the Judge: Building LLM Evaluators That Actually Work with GEPA
summary
Builds a calibrated LLM judge with GEPA using Towbench airline data (599 traces, 480 train / 112 validation), binary policy-adherence labels with human reasoning, and prompt-optimization via GEPA’s optimize-anything flow. Seed prompt + custom reflection templates and mutation/merge sampling raise validation accuracy from ~69% to ~74%, increase non-compliance recall, and reach a parto frontier with 100% per-task coverage on training. Runs take ~1 hour and cost ~$200–$300; key wins are the optimized rubric, explicit reasoning in annotations, and iterative small-batch debugging.
AI-written summary · about
chapters (15)
- 0:00LLM as judge evaluation
- 4:16Introduction
- 5:18Use case and dataset
- 8:10Metrics and annotation workflow
- 13:36Data quality challenges
- 14:39GAPA optimization algorithm
- 20:25Optimizing LLM judges with GAPA
- 22:33Jupyter notebook walkthrough
- 26:22Naive LLM judge performance
- 29:13Customizing reflection template
- 31:19Running optimization and evaluation
- 34:30Key learnings and failures
- 38:24Cost and model selection
- 39:48Best practices
- 40:31Closing and Q&A
related talks
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
32mA Practical Guide to LLM Evaluation
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
35mEvaluating LLM systems: the nuts and bolts
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
42mIs Your LLM Evaluation Missing the Point