Discovering natural language commands in multimodal interfaces
summary
A framework generates, ranks and parameterizes natural‑language command examples and ships in a prototype touch+speech image editor that shows suggestions as adaptive (tooltips), embedded (side panel) or exhaustive (command list) UIs. An online study with 24 participants (7 sessions discarded) logs 834 spoken commands from 17 valid sessions; suggestions increase speech use and discoverability, recognition errors remain common, a simple error-triggered suggestion feature proves useful, adaptive scores best and embedded elicits the most commands.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:49Discoverability challenges in speech interfaces
- 3:12Using tooltips for speech command suggestions
- 7:08Designing interfaces for command suggestions
- 10:20Framework for generating command examples
- 12:47User study setup and results
- 15:56Future work and ongoing research
- 16:18Summary and Q&A
related talks
55mDesigning for speech
59mDesigning for speech
55mDon’t Make Me Prompt: Intelligent Interfaces Users Actually Like
17mThe Voice-First AI Overlay: Designing Conversational Co-Pilots
54mConversational UX—Human Speech Patterns Meet UX
11mGestureCoach: Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations
19mFrom Demo to Deployment: Multimodal Conversational Agents
12mSensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents