ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
summary
A mobile AR authoring pipeline that combines a modified OpenMask3D scene-understanding step, automatic GPT-4 labeling, Niantic VPS localization and a fast 3D asset flow (DALL·E 2 → DIS segmentation → Instant Mesh) generates outdoor AR content in practice: offline park scans take ~13.5 minutes to produce compact scene graphs (reducing 152 noisy masks to ~12 vs ground-truth 8), on-device edits round-trip in ~30–34 seconds, asset generation runs under a minute, and a user study (N=20) produced 24 unique creations while exposing precision limits from bounding-box placement.
AI-written summary · about
chapters (8)
- 0:00Introduction to ImaginateAR
- 1:47System overview and workflow
- 2:30Scene understanding pipeline
- 5:423D asset generation process
- 7:06Mobile interface and user interaction
- 9:14User study and findings
- 12:06Limitations and future work
- 13:10Conclusion and vision
related talks
41mComputer Vision for Remote AR
15mVideo-Annotated Augmented Reality Assembly Tutorials
15mPutting the AR puzzle together with OpenStreetMap
38mDesigning Augmented Reality Systems for Speed
14mHow to AR
01h 16mAugmented Reality For The Masses
33mDesigning AR Applications
08mGesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications