Evaluation Driven Development (EDD) Explained
summary
A practical method that detects the exact failure in a multiagent chatbot where a model upgrade breaks sentiment scoring and triggers P0 escalations by tracing calls with an SDK decorator and running repeatable experiments on every change. It defines dataset, task, and metrics like policy drift and context endurance, uses LLM-as-judge then fine tunes lightweight SLM judges to score millions of traces, and treats evals plus eval engineering as CI for agents.
AI-written summary · about
chapters (11)
- 0:00Evaluation-driven development
- 2:06Traditional testing fails for AI
- 3:31What is EDD
- 5:59Challenges of EDD
- 8:25Solving indeterminism
- 10:53EDD metrics types
- 14:08Policy drift example
- 16:37Scaling EDD for production
- 19:45Eval engineering lifecycle
- 20:27Key takeaways for EDD
- 21:08Galileo platform & Q&A
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
35mThe Agentic AI Engineer
47mBuild Trustworthy AI Agents powered by Evals
49mEvals 101
36mAdvanced Evaluation Techniques for AI Agents
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
43mEvaluation is all you need
13mHow to evaluate your Evals?