Frontera: A Distributed Robot for Large-Scale Web Crawling
summary
A distributed web crawler built to scan about 600,000 .es websites to collect statistics on hosts and content sizes. Describes the crawler’s architecture and storage design, the operational problems encountered during large-scale crawling, and the solutions used to handle those issues.
AI-written summary · about
related talks
41mAddressing the Challenges of Public Web Data
14mHarnessing Common Crawl for AI and ML Applications
39mMaster Web Scraping: Become a Data Extraction Ninja!
40mWebscraping Responsibly for Social Science Research
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
18mLessons from Analysing all of Europe's Data
19mExploratory Data Science on Federated Raw Data
11mWhere Are All the Space Robots?!