The maturity phases of running evals
summary
Argues that evals should not be exhaustive unit tests but a staged practice: enumerate known failure modes, collect human justifications for thumbs up/down, scale those into LLM as judge, and close a production-to-offline flywheel of traces driving fixes. Calls out the hard engineering: context-gathering tools are manageable but CRUD tooling requires reproducing external state; two practical approaches are timestamp queries against a vector database and injecting captured system state directly into the trace.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:00What evals are and why they matter
- 6:15Four maturity phases of running evals
- 6:57Phase one: Just getting started
- 9:45Phase two: Measuring failure modes
- 12:58Phase three: Tool calls and external systems
- 16:47Future directions and advanced techniques
- 17:49Q&A: Deterministic versus LLM judges
related talks
38mBuild Evals That Actually Matter
06mWhy should anyone care about Evals?
09mHow to run Evals at Scale: Thinking beyond Accuracy or Similarity
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
46mEnhancing Agent Reliability Through Evaluation-Driven Development
17mLLM Observability, Evaluation, Experimentation Platform
49mEvals 101
38mMaking LLMs reliable - A practical framework