Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
summary
Automatic system generates concise UI phrases from mobile screenshots using a multimodal pipeline: 112k summaries across ~22k screens collected from RICO with 5 annotations per screen, text embedded with GloVe, UI structure and bounding boxes encoded, cropped element images run through ResNet, then fused into a transformer encoder-decoder to produce summaries. Model outperforms heuristic baselines on automatic metrics and MTurk ratings, supports retrieval and screen-reader/indexing uses, and dataset plus code are open-sourced.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:26SCREEN SUMMARIZATION Task formulation
- 1:01UI CORPUS
- 1:23SUMMARIZATION GUIDELINES
- 1:46LABELING INTERFACE
- 1:58SUMMARY FOCUS AREA (SFA)
- 2:18DATASET COLLECTION
- 2:43ENCODER-DECODER
- 3:00SCREEN2WORDS MODEL
- 3:52EXPERIMENTS
- 4:13RESULTS - IS DL BETTER?
- 4:33GENERATED SUMMARIES
- 4:45HUMAN EVALUATION
- 5:12LANGUAGE-BASED UI RETRIEVAL
- 5:24ENHANCING SCREEN READER
- 5:36SCREEN INDEXING FOR CONVERSATIONAL AGENT
- 6:19OPEN SOURCED DATA/CODE
related talks
22mEmpowering journalism-approaches to automatic text summarization
10mLearning Network-Based Multi-Modal Mobile User Interface Embeddings
40mThe Marvelous Magic of Multimodal AI
12mOn Suggesting Phrases vs. Predicting Words for Mobile Text Composition
19mSketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
58mAutomatic text summarization
30mChallenges & Solutions for AI-Based Message Summarization
41mSemantic Multimodal Multilingual Similarity Engine