Morae: Proactively Pausing UI Agents for User Choices
summary
Morae is a browser-extension UI agent that uses large multimodal models to detect decision points, pause, and ask concise disambiguation questions so blind and low-vision users can choose options (flavor, pickup vs delivery) instead of the agent picking the first match. It records actions, plays audio cues, and enables screen-reader navigation. A field collection of over 600 real-world tasks finds 32% contain unclear preferences; Morae yields 15% improvement on pause-required tasks and leads to more diverse, preference-aligned choices in a 10-participant study.
AI-written summary · about
chapters (8)
- 0:00Introduction to UI agents
- 1:05Current UI agent demo
- 2:29Limitations of existing agents
- 4:19Introducing Murray agent
- 6:10Proactive pausing mechanism
- 7:54Evaluation results
- 9:39Future research directions
- 11:03Summary and conclusion
related talks
36mMind Your Manners
01h 01mPracticing Accessibility Research with Confidence
13mBeyond Barriers: AI Reshaping Digital Accessibility
20mRich Representations of Visual Content for Screen Reader Users
50mEnhancing Accessibility with AI and ML
50mOur approach to accessibility at Miro, a real-time collaboration platform
22mAccessibility and Interaction in a World Beyond Screens
47mAI Systems for Accessibility