Rousillon: Scraping Distributed Hierarchical Web Data
summary
Rousillon is a programming-by-demonstration system (Helena is the readable high‑level language) that learns nested loops from a single iteration of each relation to scrape distributed hierarchical web data by replaying real DOM events. Novel relation‑selection and generalization algorithms infer which page segments form relations. A within‑subject study with 15 CS PhD students shows 8× faster authoring, 100% completion vs lower for Selenium, saving ~47–52 minutes. Production use includes ~15–20 collaborations and six deployed scrapers run in parallel; available as a Chrome extension and on GitHub as Rousillon / Helena.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:20Automate Data Collection on the Web
- 4:32Manual Collection
- 9:38User Interaction
- 10:40Relation Ambiguity
- 13:01Readability
- 13:34Reverse Compilation
- 14:06Web Scraping
- 18:49Summary of Contributions
- 20:26Not Overloading Servers
related talks
01h 01mLevelling Up Your Web Scraping Game
19mEmpowering Learners with WebR, Pyodide, and Quarto
45mThe Art of Web Scraping. Trees, Algorithms and JS Magic
05mChopping up HTML
24mimprov BOF How to Scrape Data from any Website
29mHigh Fidelity Web Crawling in Python
38mScraping LEGO for Fun: A Hacky Dive into Dynamic Data Extraction
24mDistributed Web Scraping in Python