Sprinter: Speeding Up High-Fidelity Crawling of the Modern Web
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
Sprinter combines a small browser-crawled subset with a browserless phase that replays results from a computation cache built by rewriting JS and recording input→output signatures; it uses taint tracking to limit cached state and formulates a set cover over static JS to pick the minimal browser-run pages. On a 50,000-page corpus (100 sites × 500 pages) it matches most bytes fetched by a full browser while running about 5x faster than a browser-only crawler.
AI-written summary · about
chapters (11)
- 0:00Modern web crawling challenges
- 2:22Static crawlers' failures
- 3:23Static vs dynamic crawlers
- 5:09Introducing Sprinter
- 7:36JavaScript caching reuse
- 9:44Taint tracking for caching
- 11:31Set cover for crawling
- 12:36Sprinter's workflow
- 13:39Evaluation metrics
- 15:47Scalability per site
- 16:08Conclusion
related talks
09m"Faster, Safer, More Consistent Browsers: Browserscope"
01h 01mLevelling Up Your Web Scraping Game
28mThe Browser Hackers Guide To Instantly Loading Everything
21mState of Performance
01h 01mWebcrawler
01h 03mBetter performance === greater accessibility
28mRethinking Browser Automation: Unleashing the Power of Puppeteer and Playwright
51mBenchmarking Browsers