URL Frontier, an open source API and implementation for crawl frontiers
summary
An open source crawl-frontier with a gRPC/protobuf API and Java client, queuing by key (usually host), per-URL scheduling dates, streaming get/put methods, built-in politeness and duplication checks, and a CLI plus test suite. Implementations include RocksDB, Ignite, OpenSearch implementations (single-node persisted, distributed nonreplicated, and replicated options). Tested with Stormcrawler on a virtual cluster: 354 million URLs fetched, ~1.2 billion discovered, ~37 TB of work files on S3; available as a Docker image and Maven artifacts.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:26Crawl Frontier?
- 1:07Motivation
- 2:02URL Frontier
- 4:29Main concepts
- 6:13Methods
- 7:32Basic interactions
- 9:43Command Line Interface
- 10:20Service implementations
- 12:50docker
- 13:09Large scale testing
- 13:41Hardware Sponsored by Fed Fire
- 14:39Experiment
- 16:35Current work - URLFRontier 2
- 17:40Next steps
related talks
43mFrontera: open source, large scale web crawling framework
28mFrontera: open source large-scale web crawling framework
22mOpen Web Search: A platform for a free European Web Index
01h 01mWebcrawler
45mManaging a search engine for over 600 million openly licensed media records
36mArkisto: an open-source, standards-based framework for digital preservation
06mSponsored Keynote: Foundation-Led Innovation: OpenSearch's Impact on Modern Data I...
29mWeb scraping in Elixir with Crawly