Beyond Vibes: Building Evals for Generative AI
summary
Claims an actionable evaluation workflow: a three-part approach of quick automatic metrics, sampled human preferences, and repeatable tests that run on every model change. Shows concrete checks — golden sets, sanity/unit tests for dates and totals, and cost/latency tradeoffs — and uses LM as a judge plus human rationales to scale. Demonstrates the pipeline on elevation-to–Swiss-style relief maps with parallel-coordinates comparison, a custom annotation UI, and W&B for model slicing.
AI-written summary · about
chapters (12)
- 0:00Generative AI evaluations
- 1:06Why evaluations matter
- 3:37Evals in probabilistic models
- 7:06Golden sets for testing
- 13:16Sanity checks for outputs
- 15:47Human preferences for subjective tasks
- 17:55LLMs as judges
- 20:01Error analysis and improvement
- 21:26Quality vs latency and cost
- 22:07Case study: non-text domains
- 27:46Evals pitfalls and anti-patterns
- 29:10Q&A
related talks
26m7 Habits of Highly Effective Generative AI Evaluations
20mFive hard earned lessons about Evals
49mEvals 101
43mEvaluation is all you need
46mEnhancing Agent Reliability Through Evaluation-Driven Development
09mHow to run Evals at Scale: Thinking beyond Accuracy or Similarity
47mBuild Trustworthy AI Agents powered by Evals
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work