Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
summary
An iPad-fronted system plus a Python backend uses ChatGPT‑4o to decompose a user query into a JSON step plan, Gemini 2.5 to detect 2D bounding boxes, and Segment Anything plus iPad depth to generate spatially anchored AR visuals. Manuals yield five visual categories (highlight, movement, gesture, tool, widget). Technical results report 83% plan accuracy and 65% visual accuracy, a user study N=16, and a printer-task error drop from 17% to 10% with four instructors evaluating integration.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 2:35Limitations of current AR guidance
- 3:39System contributions and approach
- 5:06Guided Reality system pipeline
- 7:14User study results and findings
- 9:47Limitations and future work
- 10:28Conclusion and Q&A
related talks
12mInteractive Planning-based Cognitive Assistance on the Edge
22mAI-Powered Personalized Learning
27mHow Foundation Models can transform interactions on visionOS
56mBeyond Chat: Bringing Models to the Canvas
18mReality and beyond
38mAI + AR Superpowers!
29mGenerative AI for Engaging User Storyboards
36mApps That See 👀 Practical AI Vision for Developers