There will be Data: Scraping the Web with Python
summary
Scrapes a static MPs directory to extract names, parties, emails and phone numbers using requests to fetch HTML and BeautifulSoup with CSS selectors (including attribute selector href^="tel:") to parse fields, producing ~450 profile dictionaries concatenated into a pandas DataFrame and stored in SQLite. Uses select_one/select and selector-gadget for robust selectors, sleeps between requests to avoid throttling, skips JavaScript-driven pages, and includes logging so breakage from site changes can be detected and fixed.
AI-written summary · about
chapters (11)
- 0:00Start
- 4:10Downsides
- 7:09Css
- 9:00Selectors
- 9:44Identifiers
- 10:21Classes
- 13:45Jupiter Notebook
- 14:22Requests
- 16:26Get Request
- 17:17Headers
- 32:58Throttling
related talks
34mWeb Scraping with Python
19mWeb Scraping for Beginners
02h 51mGathering data from the web using Python
31mBeginning your own data engineering projects
28mData Harvest: Unlocking Insights with Python Web Scraping
45mPython tools for webscraping
01h 07mWeb Scraping with BeautifulSoup & Yelp's API
32mWeb Scraping Best Practises