Testing Conversational AI
summary
Built an order-management chatbot for a US beauty brand and a test strategy that automates conversation checks, runs against Dialogflow via a service-account JSON and a Selenium-like tool (Botium) on Sauce Labs, validates responses with data-driven utterance files and intent-level assertions, and captures UI checks (buttons/images) through the same pipeline. Includes crowdtesting for regional slang, measures goal-completion, self-service and error rates, and flags regressions introduced by retraining. Demonstrates why unchecked learning can fail (Microsoft Tay) and how monitoring plus crowdsourced utterances prevent costly production mistakes.
AI-written summary · about
chapters (3)
related talks
42mTesting Conversational AI - Strategy to Automation
22mChatbot testing in the real world: Making sense of an emerging practice
56mAutomating Voice Assistants and Chatbots
17mChaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
43mTesting Agentic AI Applications: Beyond Traditional QA
34mTechnical: Harvesting Trust: Methods for Validating Chatbots in
19mTesting LLM-Powered Applications