Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations
summary
The system replaces direct selection of physical items with generated proxies so users pinch to spawn representations for nearby objects, then skim or brush across proxies for multi-selection, zoom hierarchical rows, and filter by attributes like “under $20.” Implementation uses computer vision, a multi-modal autoencoder, raycasting to the scene mesh for 3D placement, and a lazy flow that places proxies near the hand. A prototype covers books, rooms, and drone swarms and an expert study with 10 experts reports clear usability gains.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:30Introducing Reality Proxy
- 2:55Proxy interaction examples
- 4:38Application scenarios
- 6:01Proxy generation pipeline
- 7:06Expert evaluation results
- 8:09Summary and future work
- 8:30Q&A session
related talks
21mAnnexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in ...
09mAdvanced Augmented Reality interactions
16mLeaving the Reduced Reality and Into… AR, XR, VR?
14mCombining Shape-Changing Interfaces and Spatial Augmented Reality Enables Extended Object Appearance
22mAccessibility and Interaction in a World Beyond Screens
19mVirtual Replicas for Remote Assistance in Virtual and Augmented Reality
31mMerge Realities, Multiplay Wonder: Expert Guidance on Mixed Reality Development
39mDesigning for the Living Room: The Spatial Architecture of Mixed Reality