Multi-Click: Cross-Tab Web Automation via Action Generalization
summary
A browser extension called Multi-Click performs the same click or typing action across tabs by letting users pick one element, use hover+scroll generalization to expand to analogous targets, and apply actions simultaneously; it finds matches with visual cues and cross-tab XPath matching, shows color-coded preview cards for invisible pages, and uses interactive data grids for batch text entry. A user study shows it runs faster and with fewer errors than an LLM-based operator; it fails on heterogeneous sites without semantic matching.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:52Prior automation tools
- 3:37MultiClick concept
- 4:40Canvas demo overview
- 5:24Cross-tab generalization
- 6:48Real-world applications
- 7:09Generalization strategy
- 8:12User study results
- 9:15Limitations and future work
- 10:18Q&A
related talks
36mThe Pied Piper of Selenium
32mAdding Gestures to WebDriver, What a Drag
41mCross browser extensions – lets make one
26mHow browser automation works behind the scenes
30mAutonomous Browsing using Large Action Models
20mThe Anatomy of a Click
10mWhy are Web Extensions fantastic for AI?
20mXDBrowser 2.0: Semi-Automatic Generation of Cross-Device Web Page Designs