Eval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...
summary
Claims a reproducible workflow that treats every agent run as an experiment, capturing telemetry with OpenTelemetry JAI semantic convention and storing structured traces in MLflow agentic traces. Uses 20–50 daily harness runs during development and scales to 10,000+ test cases with deterministic scorers plus domain judges. Demonstrates two optimizers, Meline (fast few‑shot with vector store) and Simba optimizer (slow compile to optimized judge prompt), and a demo where a local quant model judged by Gemini improves passing rates to near 100% after alignment.
AI-written summary · about
chapters (13)
- 0:00Eval-driven dev intro
- 3:13Why agentic systems fail
- 7:03Test-driven to eval-driven
- 10:56Full trace analysis importance
- 12:41Prerequisites for eval-driven dev
- 17:40Four metrics for agentic systems
- 22:39Continuous eval loop
- 24:07Robust test harness
- 26:35Automating eval with judges
- 29:05Human feedback for learning
- 33:02Eval in CI/CD
- 34:27MLflow agentic demo
- 40:46Key takeaways
related talks
41mFrom Lab To Life: Practical AI System Evaluation
46mEnhancing Agent Reliability Through Evaluation-Driven Development
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
47mBuild Trustworthy AI Agents powered by Evals
01h 20mMind the Gap (In your Agent Observability)
31mTesting, Evaluating and Debugging Generative AI and Agentic Ap...
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge