Keynote: Reliability in AI (Agentic) Systems
summary
Study of 86 agent deployments across five domains finds failures are prevalent and no single root cause dominates; manual annotation of 150 traces plus an LLM classifier yields 95% agreement and a three‑category taxonomy: specification issues, inter‑agent misalignment, and task verification. Frameworks tested include MetaGPT, ChatDev, HyperAgent and LangChain; inserting an explicit decision step raises accuracy ~9%. Tooling named MAST and a domain‑customized Atlas pipeline convert messy logs into structured failure labels and measurably improve reliability.
AI-written summary · about
chapters (9)
- 0:00AI system reliability intro
- 3:06Prototype to production
- 7:20AI agent systems and challenges
- 10:47Failures in multi-agent systems
- 13:16Agentic system failure types
- 18:31Task verification and termination
- 20:59Custom failure taxonomies
- 23:10Taxonomies for reliability
- 24:58Summary and future directions
related talks
20mThe Three Ways AI Agents Fail. Here's How to Fix Yours.
27mHow to Build AI Agents and Keep Your Sanity
39mFrom Demo to Production: Why Agentic AI Systems Fail - and How to Fix Them
16mBuilding Applications with AI Agents
30mKeynote: Fireside Chat
34mAI Agents: Transforming Autonomous Decision-Making
46mEnhancing Agent Reliability Through Evaluation-Driven Development
27mKeynote: Towards Building Safe and Secure Agentic AI