HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based Instructions
summary
HelpViz converts web text instructions into contextual visual mobile tutorials by parsing actions, executing them on an array of Android emulators, and capturing images videos and metadata to synthesize step-by-step guides. It uses an instruction parser with beam search (top‑3 sequences) and an executor with look-ahead (up to five failed attempts per step) to improve robustness. On the PixelHelp dataset of 187 instructions the combined approach substantially increases action completion, and a 10‑participant study shows users prefer the auto‑generated contextual tutorials.
AI-written summary · about
chapters (7)
related talks
13mEmbed The Docs!
19mCodeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming
36mThe power of product screenshots in your help documentation
15mVibe Coding at Scale: Customizing AI Assistants for Enterprise Environments
27mRethinking your how-tos: From instructions to solving user problems
36mHow (and why) to make awesome video tutorials
13mReMap: Lowering the Barrier to Help-Seeking with Multimodal Search
11mNoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video U...