Vid2Coach: Transforming How-To Videos into Task Assistants
summary
System converts how-to videos into a wearable camera assistant that extracts step-level demonstration details with an LLM classifier and CLIP embeddings, augments instructions via retrieval-augmented generation on BLV resources, and monitors progress using smart-glasses video to give context-aware prompts and answers. Built prototype runs on commercial smart glasses, outperforms CLIP/Gemini for progress detection, and in an N=8 study produces 58.5% fewer errors and lower cognitive load.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 1:09How-to videos accessibility challenges
- 2:31Design insights from experts
- 3:32System demo and features
- 5:01How Vid2Coach works
- 8:14User study results
- 11:29Future directions and conclusion
related talks
22mAI-Powered Personalized Learning
45mBe My Eyes - Leveraging Generative AI to help the blind and visually impaired people
28mVigNET: An intelligent camera app that assists you...
16mVideo Personalization for Strength Training
20mNew User Interface - Part 1
15mVibe Coding at Scale: Customizing AI Assistants for Enterprise Environments
01h 23mEmpowering Learning: IT, Education, and Ethical Considerations
17mThe Voice-First AI Overlay: Designing Conversational Co-Pilots