VRSight: An AI-Driven Scene Description System to Improve Virtual Reality Accessibility for Blind...
summary
An end-to-end, post-hoc system recognizes VR scenes with object detection, depth estimation, OCR, edge-based pointer detection and LLM-driven tone selection, then delivers spatialized audio so blind users learn what is where without developer changes. Trained on DISCOVR: 17,691 social VR images across 30 object classes, the detector achieves 67.3% mAP @50 IOU, runs on consumer Quest + gaming laptop with 2 ms end-to-end latency, and a 9-participant study shows 56% seat-find success and 67% avatar-count accuracy.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 1:49Post-hoc VR accessibility solutions
- 3:13Discover dataset and object detection
- 5:45AI components in VR site
- 8:35Interaction methods and user study
- 11:24Open source and future work
related talks
47mAn Illegally Sighted Look at VR Accessibility
10mVirtual Reality platform for functional Magnetic Resonance Imaging in Ecologically valid conditions
21mEnabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory ...
41mComputer Vision for Remote AR
08mInvestigating the Effect of Sensor Data Visualization Variances in Virtual Reality
20mUsing virtual to face reality
19mVirtual Replicas for Remote Assistance in Virtual and Augmented Reality
48mTalking VR Turkey