Learning Design Semantics for Mobile Apps
summary
Automated pipeline labels Android screens with a learned lexicon of 25 UI categories, 197 text-button concepts and 135 icon classes, trained from 73k UI elements and 720 screens and applied to 72k unique Rico UIs. Code rules detect components and a CNN (CIFAR-style) classifies icons with 94% accuracy; anomaly detection drops 90% of bad icons while losing 2% valid ones, yielding 78% coverage of visible nonredundant elements.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:31Building the design lexicon
- 6:15Automated labeling pipeline
- 9:11Icon identification with deep learning
- 12:31Applications in flow identification
- 13:40Extending the approach
- 15:46Future applications and resources
- 16:08Q&A session
related talks
34mDesign System Driven Android Development: A Guide to Automation Tools
36mBuilding UI Consistent Android Apps
08mAuto-Icon: An Automated Code Generation Tool for Icon Designs Assisting In UI Development
20mDesigning for both iOS and Android—the right way
32mA Designer’s Challenge: Trying Android Engineering
39mMobile Trends 2020 - Towards a Five-Star Mobile Experience
25mGetting started on Android UI design hub
41mImprove Your App’s UI/UX By Asking The Right Questions