All Work and No Play? Conversations with a Question-and-Answer Chatbot in the Wild
summary
Deployment to 337 new hires for six weeks collects a few thousand chat utterances, labeled by a classifier into 13 conversational categories and linked to survey self‑reports; lasso regression shows four dominant interaction areas — feedback via #fail (>40% of users), chitchat about agent traits, system‑capability checks, and routine openings/acknowledgements. Reliable signals for low functional satisfaction include frequent capability queries and abrupt closings, while QA‑style lexical patterns predict satisfaction; playful users show second‑person probes and repeated casual testing.
AI-written summary · about
chapters (5)
- 0:00Start
- 2:04What Kind of Conversational Interaction Can Be Used as Signals for User Satisfaction
- 3:00Statistical Modeling
- 11:39Understanding System
- 14:02What Signals Functional Satisfaction
related talks
39mHow Chatty Leads Builds AI Agents Businesses Actually Want
22mChatbot testing in the real world: Making sense of an emerging practice
19mBigBlueBot: teaching strategies for successful human-agent interactions
35mChat Bot Building Story - The Good, the Bad and the Unexpected
17mChaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
35mDesigning Experiences for Chatbots
54mIntroduction to Chatbots
43mTesting Agentic AI Applications: Beyond Traditional QA