Evaluating AI Agents in production with Python
summary
Uploader description shows a Python-based evaluation framework that runs in production for over a year and actually detects regressions. It uses tracing with Langsmith and Phoenix (plus langchain/langraph integrations), combines LLM-as-Judge and rules-based evaluations, and computes precision/recall, semantic-similarity scores and custom factuality checks. Initial labelled sets are small—about 10 attack scenarios then 10–100 examples—and ESQL/query validation and regeneration guardrails prevent hallucinations so the system proves practical and actionable.
AI-written summary · about
chapters (11)
- 0:00AI agent evaluation intro
- 1:30Agentic AI at Elastic
- 5:21Why production evaluation matters
- 13:33Realistic test datasets
- 20:41Tracing agent decisions
- 27:17Metrics beyond precision recall
- 34:04Guardrails offline vs online
- 37:10Iterating with human feedback
- 44:21Calibrating LLM-as-a-judge
- 51:10Improving agent performance
- 57:00Q&A
related talks
33mEvaluating Your AI Agent
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
08mProduction Evals For Agentic AI Systems
37mAI agents testing How to evaluate the unpredictable
36mAdvanced Evaluation Techniques for AI Agents
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success