StreetViewAI: Making Street View Accessible Using Context-Aware Multimodal AI
summary
StreetViewAI is a working accessible Street View prototype that combines GIS data, image analysis and multimodal Gemini models (Gemini Flash for structured JSON descriptions and Gemini Live for real‑time chat/commands) to let blind users query and navigate 360° imagery across Google Street View’s 220+ billion images. The system exposes AI Describer and AI Chat with bi‑directional semantic commands, was evaluated with 11 blind participants (816 questions, 85% correct answers, chat used ~7× more than describer) and scores 6.4/7 usefulness. Error types: misidentifications, false negatives, out‑of‑view limits.
AI-written summary · about
chapters (11)
- 0:00Introduction and welcome
- 2:01Street View accessibility challenges
- 3:26Street Reader AI prototype demo
- 5:11User tasks and system design
- 7:58AI Chat and real-time interaction
- 8:40User study with blind participants
- 10:26Key findings and limitations
- 11:09Common user questions analysis
- 12:30AI accuracy and error types
- 13:52Future work and agentic AI
- 14:35Closing remarks and potential
related talks
50mAI as an A11y Ally
46mArtificial intelligence to the rescue of accessibility
34mAI and Accessibility: the Good, the Bad, and the Bollocks
30mAI and Accessibility: the Good, the Bad, and the Bollocks
48mAI and Disability: Progress or Exclusion?
13mConnecting data sources with AI
28mVigNET: An intelligent camera app that assists you...
46mBuilding accessible experiences with AI