Evals 101
summary
Walks through building end-to-end evals on Braintrust: define a task (prompt or agent), push datasets and scores (code or LLM-as-judge), run offline experiments and CI checks, then instrument production with the SDK wrappers to capture spans and enable online scoring with sampling and regression alerts. Shares tools: Python/TypeScript SDKs, auto evals package and a new loop optimizer; recommends high-quality models for scoring, Levenshtein as a quick baseline, and human review to add production failures back into datasets.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:51Company overview
- 3:14Why use evals?
- 5:30Core concepts
- 6:52Components of an eval
- 8:31Offline vs online evals
- 10:38Prompt and data design
- 12:50Start small and iterate
- 14:32Types of scorers
- 16:09Playgrounds and experiments
- 18:24Evals via SDK
- 26:15Moving to production
- 30:51Online scoring
- 33:25Custom views
- 36:07Human in the loop
- 38:53Closing the loop
- 40:51Q&A
related talks
43mEvaluation is all you need
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
31mBeyond Vibes: Building Evals for Generative AI
47mBuild Trustworthy AI Agents powered by Evals
38mBuild Evals That Actually Matter
24mHow to Test AI Features Before Your Users Do
32mEvaluating multi-turn conversations