Testing Conversational AI - Strategy to Automation
summary
Begins with the Microsoft Tay disaster (shut down within 24 hours) and shows concrete failures: CNN alerts that keep sending, weather bot missing “weekend” intent, 15 automated test categories with 2 failures in a demo. Uses Node/npm, Mocha and an open-source suite with Botium CLI plus a UI Botium Box, pushes tests into CI/CD, captures utterances via crowd testing, analyzes a confusion matrix to measure precision/recall and decide whether to retrain models or fix integration bugs.
AI-written summary · about
chapters (14)
- 0:00Start
- 3:14Self Learning Systems
- 3:56No barrier
- 8:22How do you test this?
- 9:02Test Strategy
- 9:42Personality and on-boarding
- 11:31Intelligence
- 12:49Navigation
- 18:56Botium in nutshell
- 23:39Writing Tests
- 23:57Single Conversation Flow
- 25:16With Utterance Files 1
- 26:21With User Interface Elements
- 28:19What's Next?
related talks
47mTesting Conversational AI
43mTesting Agentic AI Applications: Beyond Traditional QA
17mChaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
22mChatbot testing in the real world: Making sense of an emerging practice
15mPractical tactics to build reliable AI apps
56mAutomating Voice Assistants and Chatbots
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
19mTesting LLM-Powered Applications