Distributed Web Scraping in Python
summary
Builds a controller + work and result queues feeding many scraper nodes that use requests + BeautifulSoup + Selenium, proxy rotation, random headers and retries to process 25,000 pages workloads. Uses message brokers like RabbitMQ/Redis/Kafka with pika/puka for dispatch, plus tooling like click retry backoff requests-cache and fake-user-agent. Empirical result: a 15k–25k job goes from about 26 hours reduced to 4–5 hours on 8–10 nodes, at the cost of higher resource and deployment complexity.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:39Outline
- 1:03Introduction
- 2:15Data Science Project Stages
- 3:04What is Distributed Web Scraping
- 4:13Setting the Stage
- 5:26Iteration - A Single Request
- 6:07Looping Requests
- 6:46Iteration 1 - Issues
- 8:08Intermediate improvements
- 10:39Iteration 2 - Issues
- 11:42Distributed - Mental Model
- 12:15Distributed - Controller
- 12:47Distributed - Scraping Node
- 14:05Distributed - Advantages
- 15:19Distributed - Disadvantages
- 16:07Distributed - Queues
- 17:04Distributed - Message Brokers
- 17:54Code Management
- 19:54Useful Python Packages
- 21:52Considerations
- 22:49Conclusion
related talks
22mDistributed Web Scraping in Python
28mBoosting Python Web Apps performance with Celery and Concurrency
29mHigh Fidelity Web Crawling in Python
29mDive into Scrapy
32mWeb Scraping Best Practises
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
19mOptimizing Data Retrieval with Python Celery
02h 51mGathering data from the web using Python