Addressing the Challenges of Public Web Data
summary
Common Crawl maintains a 10 PB archive of about 30 billion pages, adding ~2.5 billion pages a month and serving nearly 1 exabyte of downloads last year, and uses crawl metadata, a host index and a web graph to measure exclusions from robots.txt, legal opt-outs and new bot defenses. Measurements show blocking spikes after Nov 2022 concentrate on top sites so training corpora lose head material but overall model quality is not yet decimated. New metadata explorer and a Neo4j web graph conversion are being rolled out. 10 PB, 30B pages 2.5B pages/month 1 exabyte downloads last year
AI-written summary · about
chapters (21)
- 0:00Common Crawl mission
- 1:05Exponential growth and impact
- 3:10Linguistic and cultural diversity
- 6:37Polite web crawling
- 8:24AI panic and web blocking
- 10:52Impact of AI crawler bans
- 13:43Sector-specific web blocking
- 14:51AI training data quality
- 16:15Legal and policy challenges
- 17:39Opt-out requests surge
- 19:03Language distribution gaps
- 20:51Crowdsourcing coverage
- 22:37Q&A on AI data
- 25:45Language trends and models
- 28:56Crawling restricted regions
- 29:18LLM training data sources
- 31:12Structured data products
- 33:19AI panic’s limited impact
- 37:09Crawling YouTube
- 38:13Future impact on AI
- 39:59Roadmap for new products
related talks
14mHarnessing Common Crawl for AI and ML Applications
50mHow We Crawl 60 Million Web Pages a Day
43mData & Metadata at the Internet Archive
06mFrom RUM to Robot Crawl Experience
13mThe silent infrastructure crisis
17mDecentralized Technologies at the Internet Archive
47mFrontera: A Distributed Robot for Large-Scale Web Crawling
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites