Frontera: open source, large scale web crawling framework
summary
A scalable crawler framework is built and released as Frontera, using Scrapy, Kafka and HBase with per‑host queues, local state cache and DNSmasq; the team aims to sweep ~600K .es sites but actually discovers ~60K domains and fetches 46M pages in 1.5 months, storing ~3 TB compressed (~275 bytes/doc). Key fixes include host‑local HBase keying, per‑host metering to avoid floods, message compression, and a limit of 200 pages per host; an online scheduling model lets strategies change without stopping the crawl.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 4:16Scrapinghub and Frontera
- 10:13Testing Frontera Spanish crawl
- 12:52Frontera architecture
- 16:38Big and small host problem
- 18:53DNS resolution optimization
- 21:02HBase bottlenecks
- 23:32Crawling results insights
- 29:41Deployment requirements
- 35:17Frontera features
- 37:46Customization and support
- 40:41Future plans and contribution
- 41:24Scrapinghub cloud services
related talks
28mFrontera: open source large-scale web crawling framework
21mURL Frontier, an open source API and implementation for crawl frontiers
29mWeb scraping in Elixir with Crawly
22mOpen Web Search: A platform for a free European Web Index
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
50mHow We Crawl 60 Million Web Pages a Day
01h 01mWebcrawler
29mDive into Scrapy