Chaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
summary
Built an AI simulation framework that evaluates multi‑turn customer journeys rather than single prompts, using a customer simulator fed by seed questions from golden or synthetic datasets and a YAML world model. Inputs are persona and ontology, task and end state, and the loop updates state until success or max turns. Metrics run per turn for correctness, task completion, recovery, consistency and tool‑usage accuracy, revealing failures like wrong tool use, premature answers, policy violations, drift and “confidently wrong” resolutions even across 20+ turn conversations and enabling regression comparisons between candidate and control models.
AI-written summary · about
chapters (7)
- 0:00Introduction to chaos testing
- 3:23Simulating real customer behavior
- 6:32Persona and ontology design
- 10:23Simulation flow and metrics
- 12:48Traditional vs realistic evaluation
- 14:11Demo of customer simulator
- 16:42Why realistic evaluation matters
related talks
43mTesting Agentic AI Applications: Beyond Traditional QA
47mTesting Conversational AI
22mChatbot testing in the real world: Making sense of an emerging practice
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
15mPractical tactics to build reliable AI apps
42mTesting Conversational AI - Strategy to Automation
31mIntroducing Agent Testing
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems