[Evals Workshop] Mastering AI Evaluation: From Playground to Production
summary
Hands-on workshop shows how to run evals from playground to production using Braintrust: push prompts and datasets from a repo with pnpm/braintrust push, run offline AB tests in the playground, and track experiments over time. Uses OpenAI under the hood, supports tools and RAG, and scores with LM-as-judge and deterministic code enrollments (accuracy, formatting, completeness). Demonstrates logging wrappers for live traces, online scoring with sampling and alerts, and feeding bad/good spans into datasets for human review.
AI-written summary · about
chapters (14)
- 0:00Introduction and agenda
- 2:58Why use evals?
- 6:31Core Braintrust concepts
- 10:24Deep dive into tasks
- 13:37Datasets and scorers
- 15:14Playgrounds vs experiments
- 17:41UI workshop activity
- 20:41Technical setup and push
- 26:13Playground activity walkthrough
- 37:46SDK and version control
- 40:00Debugging and QA
- 55:23Production logging and online scoring
- 1:00:45Live app activity
- 1:05:18Human in the loop and Q&A
related talks
01h 26mShipping AI That Works: An Evaluation Framework for PMs
47mBuild Trustworthy AI Agents powered by Evals
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
24mHow to Test AI Features Before Your Users Do
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
41mFrom Lab To Life: Practical AI System Evaluation
32mEvaluating multi-turn conversations
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability