Will LLMs really consume the Internet?
summary
Claim: the largest LLMs get trained and fed by massive web scraping operations that deliver 2.5 petabytes a day from ~28 billion requests, about 500 million pages. Tools are concrete: proxies for geo IPs, browser fingerprinting, browser automation, captcha solving and a commercial web unlocker stack. Surprising facts: legal precedent wins vs Meta/Twitter, agents act like targeted crawlers, some agents already order food on real sites but still hit blocks (Amazon example).
AI-written summary · about
chapters (12)
- 0:00AI and web scraping intro
- 2:27Bright Data’s public web data
- 4:37AI agents need internet
- 6:22AI’s data hunger scale
- 8:50Web blocking and captchas
- 12:01Reliable content access
- 15:39Solving captchas and automation
- 16:43Parsing scraped HTML
- 17:48Ethical scraping concerns
- 20:36Robust web agent solutions
- 21:17AI food ordering demo
- 23:24Summary
related talks
10mThe Impact of LLMs on the Circular Economy and Beyond
01h 10mTeach an LLM to Navigate the Web
49mLLM—Search Engine Killer?
28mAre LLMs the answer to all our problems?
37mExpanding the Capabilities of LLMs Through Community Collaboration
17mWebLLM: A high-performance in-browser LLM Inference engine
35mReimagine financial service experience with LLM
46mLLM—Search Engine Killer?