How to Test AI Features Before Your Users Do
summary
A lightweight eval loop is built around an anonymized knowledge assistant in Microsoft Foundry that catches three concrete failures: unsupported claims, missing key information, and brittle behavior after prompt or model changes. The demo uses a compact golden dataset (start with ~10 rows), runs scheduled app-evals, and outputs rubric scores (pass/fail, 1–5, or pairwise) so teams can trace failures to retrieval, generation, or tooling and use LLM-as-judge to automate grading and regressions.
AI-written summary · about
chapters (12)
- 0:00Testing AI features intro
- 3:17Why AI needs rigorous tests
- 6:11Unit tests vs evaluations
- 8:18Benchmarks versus application evaluations
- 12:09Test cases for AI agents
- 13:57Evaluators in Foundry
- 16:08Setup with golden datasets
- 18:40Analyzing results
- 20:49Scoring methods
- 21:50Demos aren’t enough
- 22:37AI failure patterns
- 23:20Q&A
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
22mMaking AI Accountable: Azure AI Foundry
15mPractical tactics to build reliable AI apps
01h 26mShipping AI That Works: An Evaluation Framework for PMs
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
38mAI Just Happened to Me : Help!!
25mAI vs. AI: How to Test AI Systems Using AI
47mBuild Trustworthy AI Agents powered by Evals