Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK
summary
Built an agent with the AI toolkit for VS Code that uses Playwright to scrape event pages, extract name/date/location and attendee count (demo shows 269) by combining Reactor and Luma pages; compares models (GPT-4.1 vs 4.0) and finds 4.1 much faster and preferred. Uses Azure AI Evaluation SDK / Azure Foundry to run batched, programmatic evaluators (relevance, coherence, groundedness, fluency plus F1/BLEU/ROUGE), exports JSONL, and reports per-item scores on a 1–5 scale with configurable thresholds.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:30Overview
- 1:23AI agents are all the rage
- 2:12How do you evaluate AI agents
- 2:40When to evaluate AI agents
- 3:26How to evaluate AI agents
- 4:15Manual model evaluation
- 7:10Evaluating the whole system
- 14:33Scale
- 16:59Multimodel Models
- 18:46Outro
related talks
22mMaking AI Accountable: Azure AI Foundry
25mBuilding AI Agents
39mEvolving Intelligence: Designing Autonomous AI Agents
58mAdvanced Autonomous AI Agents: Memory, Collaboration and Evolution in Action
36mBuilding AI Agents in .NET with Azure AI Foundry
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
38mSafe Evaluation and Rollout of AI Models
56mBuilding AI Agents and Multi-Agent Systems with Azure AI