Your AI Agent Has No SLO
summary
Built an agent observability stack that wires agent traces and attributes into OpenTelemetry → Prometheus → Grafana, defines SLOs and error budgets, and alerts on reasoning gaps. Introduces five agent-specific SLIs: calibration error, evidence completeness, data currency, contradiction rate, and decision tier distribution. Production example: an agent marks 23 Medicare claims 80–91% confident but 19 of 23 are wrong; alerts fire and processing is stopped, preventing costly denials.
AI-written summary · about
chapters (12)
- 0:00Introduction to AI agents
- 1:12Observability challenges in AI agents
- 3:00Why traditional SLOs fail for agents
- 5:34Common agent failure modes
- 8:05Nine SLO principles for agents
- 10:14Tiered validation framework
- 11:43Error budget example with claims
- 13:51Limitations of AI agents in production
- 15:15Grafana dashboard walkthrough
- 17:41Token usage and hallucination monitoring
- 19:07Reasoning chain and SLO metrics
- 20:33Conclusion and future work
related talks
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
27mWhy Are Your AI’s Decisions Hard To Explain: Trace Every Decis...
30mTutorial: Building an Observable AI Agent With an Open Source Stack
30mAI Agents for Incident Management
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
31mDeep Dive To AI Agent Observability
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry