Learning Network-Based Multi-Modal Mobile User Interface Embeddings
summary
Frames mobile UIs as a bipartite screen–element graph and learns joint embeddings that reconstruct adjacency and attributes, using per-modality GAT encoders, inter-modality attention, a second GCN-based VAE stage, and MMD instead of KL for stability. Trained and evaluated on RICO-N/RICO-O with link prediction, ratings regression, attribute inference and retrieval (Mechanical Turk), the model MAAN outperforms GAT, CAN and VSE++ baselines across tasks by a noticeable margin and shows clearer clustering in embedding space.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 1:32Model overview and contributions
- 4:57Model architecture details
- 5:42Dataset and experimental setup
- 7:11Results and performance comparison
- 9:51Conclusion and future work
related talks
07mScreen2Words: Automatic Mobile UI Summarization with Multimodal Learning
40mThe Marvelous Magic of Multimodal AI
13mTraining Embedding Model Resiliently for Multimodal M...
56mApplications of Multimodal Learning in Media Search Engines
16mAre Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership...
24mMulti-Modal AI Agents Explained | Open-Vocabulary Alignment
41mSemantic Multimodal Multilingual Similarity Engine
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently