Distributed Web Scraping in Python
summary
Builds a Python distributed scraper that cuts a 15,000-page job from about 26 hours to 4–5 hours by running many workers in parallel; uses a two‑phase controller that generates URL queues, worker nodes that use Selenium headless browser plus rotating headers, delays and proxies, and a results consumer that validates and stores data; communicates via message brokers like RabbitMQ + pika (Redis also possible) and deploys workers on DigitalOcean droplets; system works but raises cost, redeploy and trust/security issues.
AI-written summary · about
chapters (9)
- 0:00Introduction to speaker
- 0:53Data pipeline responsibilities
- 2:57Basics of web scraping
- 4:49Limitations of simple scripts
- 7:34Iterative scraping solutions
- 10:59Distributed scraping architecture
- 14:39Message brokers and queues
- 17:30Advantages and disadvantages
- 20:47Future developments and closing
related talks
24mDistributed Web Scraping in Python
28mBoosting Python Web Apps performance with Celery and Concurrency
29mHigh Fidelity Web Crawling in Python
29mDive into Scrapy
32mWeb Scraping Best Practises
02h 51mGathering data from the web using Python
31mBeginning your own data engineering projects
28mData Harvest: Unlocking Insights with Python Web Scraping