Shipping AI That Works: An Evaluation Framework for PMs
summary
Concrete claim: a reproducible eval loop (dataset → prompt playground → experiment → LLM-as-judge → human labels) surfaces real gaps. Demo builds a multi-agent trip planner with Langraph and Cursor, logs traces into Arise / open-source Phoenix, runs an A/B on 12 examples and auto-evals; adding a single prompt line yields a 100% “offer discount” label but the judge often disagrees with human annotations, and the live prototype intermittently fails — evidence that tooling, metrics and human-in-the-loop labels actually decide shipability.
AI-written summary · about
chapters (13)
- 0:00AI product evaluation intro
- 3:22PMs need eval frameworks
- 7:53Eval vs software testing
- 13:33Live demo: AI trip planner
- 18:53Agent traces and visualization
- 23:24Iterating prompts with data
- 30:54Scaling evaluations
- 38:14Tone and offers evaluation
- 45:02Building eval prompts
- 53:48Dev to production workflow
- 1:01:29Q&A on eval teams
- 1:17:14Future of AI teams
- 1:24:08Closing remarks
related talks
33mEvaluating Your AI Agent
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
36mAdvanced Evaluation Techniques for AI Agents
24mHow to Test AI Features Before Your Users Do
15mPractical tactics to build reliable AI apps
41mFrom Lab To Life: Practical AI System Evaluation
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
32mEvaluating multi-turn conversations