Dive into Scrapy

on YouTube ↗29m

" summary

Claims a production-ready Scrapy stack that scales to millions of requests and is used at Scrapinghub, built on Twisted with a scheduler, downloader, middlewares and item pipelines that export JSON/CSV/XML and map to Django models. Shows concrete tools: Scrapyd and Scrapinghub cloud for deployment, the interactive shell and item loaders for field parsing, pipelines for validation and storage, and Splash headless WebKit for JS pages. Recommends concrete anti-blocking tactics: rotate user agents, disable cookies, randomize delays and use rotating proxies.

AI-written summary · about

" chapters (23)

1. 0:00Intro
2. 0:29API
3. 4:30Python
4. 5:57HTML Parsers
5. 6:46Example
6. 8:27What I recommend
7. 9:07Interactive Cell
8. 10:11XKB Project
9. 13:29Item Loader
10. 14:08Item exporters
11. 14:37Jungle Item
12. 14:57Under the hood
13. 15:53Item Pipeline
14. 17:25Item Requests
15. 18:25Escapee
16. 19:14Visualization
17. 20:01How to avoid getting banned
18. 21:45Open Source
19. 22:27CreepyDemon
20. 23:08EscapeeCloud
21. 24:28About our products
22. 24:49Hiring
23. 25:07QA

" related talks

• 46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!Fabien Vauchelles · DevDays Europe 2024
• 45mPython tools for webscrapingJose Manuel Ortega · PyData Madrid 2016
• 24mDistributed Web Scraping in PythonJosh Weissbock · PyCon US 2022
• 02h 51mGathering data from the web using PythonRenne Rocha · PyCon US 2024
• 22mDistributed Web Scraping in PythonJosh Weissbock · PyCascades 2019
• 28mBoosting Python Web Apps performance with Celery and ConcurrencyYash Raj · PyCon Italia 2025
• 28mData Harvest: Unlocking Insights with Python Web ScrapingYuliia Barabash · PyData Global 2023
• 01h 01mLevelling Up Your Web Scraping GameIan Littman · PHP UK Conference 2022