Frontera: open source large-scale web crawling framework
summary
Frontera is an open source, real‑time web crawling framework that implements single‑threaded and distributed modes and exposes canonicalization, scoring and queue backends. It demonstrates a working pipeline built with Scrapy + Kafka + HBase, uses partitioning to be polite by partitioning, and requires HBase and Kafka for distributed operation. A Spanish crawl runs 12 spiders for 1.5 months and yields 47 million pages, showing the system actually scales and produces measurable results.
AI-written summary · about
chapters (19)
- 0:00About me
- 1:19What is Frontera
- 1:58What is Terra
- 2:29Motivation
- 4:50Single threaded
- 5:58Single integration
- 8:16Unique content
- 9:09Metadata storage
- 9:57Architecture
- 11:39Scrapping
- 12:43Simple spider
- 13:24Use cases
- 15:02Architecture distributed
- 17:48Features
- 18:56Requirements
- 19:51Quick start
- 20:29Spanish crawl
- 21:37Future plans
- 22:29Questions
related talks
43mFrontera: open source, large scale web crawling framework
21mURL Frontier, an open source API and implementation for crawl frontiers
29mWeb scraping in Elixir with Crawly
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
24mDistributed Web Scraping in Python
22mDistributed Web Scraping in Python
01h 01mWebcrawler
29mUn Enfoque Pythonico para Minería de Textos de Historias de Trabajo Arduo, Migración y Resiliencia