Ship Real Agents: Hands-On Evals for Agentic Applications
summary
Builds a complete eval pipeline for a financial‑analysis agent that actually shipped: instrumenting with Phoenix tracing, running the agent with the Claude agent SDK, categorizing failures from traces, then adding deterministic code checks and LLM-as-a-judge evals (Anthropic) plus a custom rubric and labeled examples. Key data point: a correctness judge returns 0 out of 13 correctness while a faithfulness judge returns 13 out of 13 faithfulness, and experiments prove prompt changes either fix or regress behavior.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:14Workshop Overview
- 4:31Troubleshooting Phoenix Setup
- 5:17Fundamentals of Evals and Tracing
- 18:44Anatomy of an Eval Result
- 21:19The Iteration Loop
- 26:58Building the Financial Analysis Agent
- 33:28Using Phoenix for Observability
- 35:38Running Multiple Test Queries
- 38:12Reading and Categorizing Traces
- 49:52Implementing Code Evals
- 57:51Built-in LLM-as-a-Judge Evals
- 1:03:04Faithfulness Evaluation
- 1:04:35Designing a Custom Eval Rubric
- 1:11:47Running the Actionability Judge
- 1:19:14Using Data Sets and Experiments
- 1:50:19Final Tips and Best Practices
- 1:51:48Differences Between Phoenix and Arize AX
related talks
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
35mThe Agentic AI Engineer
01h 26mShipping AI That Works: An Evaluation Framework for PMs
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
54mYou Can't assertEquals an Agent: A Tester's Guide to Agentic Quality
37mAI agents testing How to evaluate the unpredictable