Mind the Gap (In your Agent Observability)
summary
Claims a working demo: the observe skill demo points at a live Foundry agent, auto-generates an eval dataset, runs batch evaluations, optimizes the prompt, compares versions and rolls back to the best one, all while logging its chain-of-thought. Instrumentation uses tracing built on OpenTelemetry with App Insights/Azure Monitor, workflow agents emit trace-linked evaluations, and red teaming runs adversarial prompts. Watch for the concrete numbers of failing cases, the exact evaluator names, the model and tooling swaps it tries, and the version that it chooses to keep.
AI-written summary · about
chapters (10)
- 0:00Agent observability gap
- 14:20Workshop setup and use case
- 23:03Travel agent in Microsoft Foundry
- 34:09Tracing and evaluations in agent workflows
- 40:09Building agents with SDK and workflows
- 51:11Custom tracing and Azure Monitor
- 55:03Agent quality and safety metrics
- 1:00:18Red teaming agent security
- 1:04:07Automating observability
- 1:16:04Closing remarks
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
43mEval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
35mThe Agentic AI Engineer
29mYour Prompt Is Not a Unit Test — instrumenting LLM apps in production