High Fidelity Web Crawling in Python
summary
Shows concrete techniques: use Python Requests for simple pages and Selenium + headless browsers (PhantomJS now moving to Chromium) for JavaScript, add logging and alerts, use expected-conditions and retry loops to recover from timeouts, and throttle via random sleeps. Runs 100,000 pages weekly and sees runtime jump 2 to 72 hours when polite proxies and delays are used; rotate a few proxies (2–5) and randomize from a large list like 900 user-agent headers. Applied to hockey player pages and yields usable data for scouting and drafting.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:05Python scraping needs robustness
- 4:56Basic scraping example
- 6:01Simple scripts fail in production
- 7:08Logging and alerting
- 8:12Dynamic JavaScript sites
- 10:40Waiting for JavaScript
- 12:30Retrying failed requests
- 13:54Scraper blocking and ethics
- 14:57Proxy servers
- 16:44Randomizing proxies and agents
- 17:48Throttling requests
- 19:54Speed versus anti-bot
- 20:15Three lessons for production
- 22:04Q&A
