Ship Real Agents: Hands-On Evals for Agentic Applications
summary
Builds a financial‑analysis agent, instruments it with trace with Phoenix, and shows exact checks: simple code evals, built‑in and LLM‑as‑a‑judge rubrics, and a labeled custom rubric with meta‑evaluation. Demonstrates a startling contrast — faithfulness eval 13/13 vs correctness 0/13 — because the judge choice matters. Runs 12 test queries, recommends 200–400 samples to ship, and runs focused experiments that prove a prompt change actually improves actionability rather than relying on vibes.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:14Workshop Overview
- 4:31Troubleshooting Phoenix Setup
- 5:17Fundamentals of Evals and Tracing
- 18:44Anatomy of an Eval Result
- 21:19The Iteration Loop
- 26:58Building the Financial Analysis Agent
- 33:28Using Phoenix for Observability
- 35:38Running Multiple Test Queries
- 38:12Reading and Categorizing Traces
- 49:52Implementing Code Evals
- 57:51Built-in LLM-as-a-Judge Evals
- 1:03:04Faithfulness Evaluation
- 1:04:35Designing a Custom Eval Rubric
- 1:11:47Running the Actionability Judge
- 1:19:14Using Data Sets and Experiments
- 1:50:19Final Tips and Best Practices
- 1:51:48Differences Between Phoenix and Arize AX
related talks
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
37mAI agents testing How to evaluate the unpredictable
33mEvaluating Your AI Agent
54mYou Can't assertEquals an Agent: A Tester's Guide to Agentic Quality
35mThe Agentic AI Engineer
36mAdvanced Evaluation Techniques for AI Agents