AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information ...
summary
AROMA is a working mixed-initiative system that grounds video recipes to reality by combining a wearable chest-mounted camera, non-visual user cues (touch, smell, taste), and a machine-learning pipeline that parses videos into JSON and triggers a finite state machine plus a backend LLM. It implements four response types (food state, step-related, problem-solving, visual questions) and was tested with eight blind participants who cooked unfamiliar recipes; results show timely proactive alerts, useful segmented replays, and improved task mapping and usability.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 1:51How AROMA works
- 3:37Demo and response types
- 4:41User study and results
- 6:50Design challenges and future work
- 8:56Q&A
related talks
40mA Co-Design approach for automated cooking appliances
23mConnecting Food Journeys through AI and the Open Food Platform
13mInteractive Learning and User‑Centered AI: A Retrospective
27mmulti-modal AI to avoid eating takeaways
43mAI w/o Spaghetti: Clean Architecture in the Age of AI
50mAI as an A11y Ally
22mLearning How to Learn in the AI Era
37mAI and Emotions: opportunities and challenges (Virtual Talk)