Gathering data from the web using Python
summary
Covers a hands‑on Scrapy tutorial: building spiders, pagination and exporting with the JSONL vs JSON memory tradeoff, handling AJAX by calling the site’s underlying API, extracting data embedded in script tags with regex + json loads, and optionally using headless browsers (Playwright/Selenium) via scrapy-playwright; exercises scrape a quotes sandbox, produce 100 items, demonstrate caching, rate control, proxies and Spidermon monitoring, and show when a browser is necessary versus parsing API responses.
AI-written summary · about
chapters (14)
- 0:00Web scraping intro
- 23:47Python setup for Scrapy
- 56:15Why Scrapy is best
- 1:02:36Exercise 1: static pages
- 1:30:01Break & Q&A
- 1:31:04Logging into sites
- 1:32:10Exercise 2: infinite scroll
- 1:59:51JS variable scraping
- 2:11:08Exercise 4: multiple requests
- 2:25:33Solution review 4
- 2:31:44Headless browsers intro
- 2:41:56Long-term scraper maintenance
- 2:47:01Legal & ethical scraping
- 2:50:15Closing & Q&A
related talks
24mimprov BOF How to Scrape Data from any Website
45mPython tools for webscraping
28mData Harvest: Unlocking Insights with Python Web Scraping
34mWeb Scraping with Python
35mLittle Data Soup with Spinach (Web Scraping)
29mHigh Fidelity Web Crawling in Python
32mWeb Scraping Best Practises
19mWeb Scraping for Beginners