StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
summary
StateLens reverse engineers on-device UI structure by mining POV and web videos and building state diagrams with a hybrid crowd-computer vision pipeline (screen detection, SURF, OCR) and finger-detection, then auto-generates a conversational task agent via Google Dialogflow and an iOS guidance app with head-mounted camera. Risk-free exploration uses 3D-printed accessories (finger ring, cap, stylus) to avoid accidental touches. Technical evaluation on 12 devices and >70,000 frames reconstructs interfaces accurately, and a user study with 14 blind participants shows the system enables access to previously inaccessible touchscreens.
AI-written summary · about
chapters (11)
- 0:00Introduction and motivation
- 1:49Challenges with dynamic touchscreens
- 3:35StateLens system overview
- 5:45Building state diagrams from videos
- 8:31Conversational agent for task specification
- 9:55Real-time state identification
- 10:563D printed accessories for exploration
- 11:59Evaluations and user study results
- 13:04Future hardware solutions
- 14:31Broader applications and summary
- 15:33Q&A session
related talks
21mVizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
58mAccessible Spatial Computing
09mSmart Glasses for the blind
25mAlternative Smartphone Interactions based on Sensor Data
29mCan't touch this - Different ways users interact with their mobile devices
20mNew User Interface - Part 1
14mTransform intent into action through your eyes
09mScreen Parsing: Towards Reverse Engineering of UI Models from Screenshots