[Full Workshop] Building Metrics that actually work
summary
Workshop shows a concrete scoring system: start with 5–10 simple signals then iterate to hundreds (Google Search used ~300) and calibrate against a thumbs‑up/thumbs‑down ground truth set (~120 examples). Tools provided: a Co‑pilot that generates examples, a Colab SDK for batch runs and model comparison (1.5 vs 2.5), and a Sheets integration to copy criteria into Google Sheets and view confusion matrices. Scorers are deterministic, use bidirectional models with a regression head and score in sub-50ms, and the practical trick is to crank up temperature and rank multiple outputs to get an immediate quality lift.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:44Whats the correct answer
- 14:02Building nuanced evals
- 16:17Scoring system
- 19:07Demo
- 20:32Reasoning Model
- 28:17Building a Criteria
- 29:07Example Criteria
- 32:27Model Architecture
- 36:17Collab
related talks
37mOptimal on Paper, Broken in Reality
20mAgentic Evaluations at Scale, For Everybody
32mEvaluating multi-turn conversations
31mBeyond Vibes: Building Evals for Generative AI
12mGamify the Metrics, Amplify the Impact
27mTeam Assessments: the Good, the Bad and the Ugly
30mBuilding a Robust Evaluation Framework for GenAI Productivity Tools
45mThe Definitive Answer to Measuring Developer Productivity