Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initi...
summary
PrISM is a multimodal assistant that trains a smartwatch-based step tracker from a single demonstration using event detection and an LLM-generated task transition graph, then corrects sensing errors through mixed-initiative dialogue where an LLM pipeline classifies utterances to update step hypotheses. A 20-participant study on a 19-step espresso task raises step-tracking accuracy from 45% to 71.1% and cuts inappropriate responses from 27% to 9.9%; ongoing month-long deployments involve over 100 patients.
AI-written summary · about
chapters (7)
- 0:00Introduction to context-aware assistants
- 1:49Challenges in smartwatch sensing
- 3:34Mixed-initiative dialogue approach
- 7:24Demo of adaptive context updates
- 8:29User study results and impact
- 9:11Framework deployment and future work
- 9:52Q&A on step granularity
related talks
12mInteractive Planning-based Cognitive Assistance on the Edge
10mApp-Based Task Shortcuts for Virtual Assistants
19mFrom Demo to Deployment: Multimodal Conversational Agents
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI
22mAI-Powered Personalized Learning
26mMastering Large Language Models: Prompt Engineering for Precision & Performance
03mAutonomy and Human-AI Interaction
10mA Day in the Life: An Immersive Data Experience