Scraping Your Way to a Dataset
summary
Shows how to turn two messy sources into usable datasets by actually running requests and BeautifulSoup on the PA Lottery pages (iterating roughly 72,000 consecutive IDs to find Powerball draws), reverse‑engineering a university CSRF token + POST endpoint to pull class sections, and saving raw HTML to avoid re‑hits. Code includes simple async ideas (httpx/requests‑futures), CSV export plus tidy data normalization into courses and sections tables, and a basic randomness test that meets the expected threshold.
AI-written summary · about
chapters (23)
- 0:00Start
- 1:07Organizing data into a dataset
- 2:06What data is needed?
- 3:00Consider the following problems...
- 5:18The dataset doesn't exist
- 6:28Scraping is a last resort
- 6:39Potential Issues with Web Scraping
- 9:06Back to the program
- 12:13The Developer Console
- 13:52Probably calling a Web API
- 16:53Web API can be fun!
- 17:49Data scraping exploration phase
- 19:30Into the world of BeautifulSoup
- 30:00Stop spamming web servers
- 30:21Limit your scraper
- 31:12Preventing future requests
- 32:20Try scraping publicly available information
- 32:58These datasets are bad for analysis
- 35:17Closely related to Database Normalization
- 35:30Data Integrity
- 36:39Let's Apply These Principles
- 40:50We reached the end of journey
- 41:29Need something a little more powerful?
related talks
40mWebscraping Responsibly for Social Science Research
31mBeginning your own data engineering projects
22mService Dogs in the Cabin - Web Scraping Puppy Data with Microsoft Fabric - 1st Leg
22mService Dogs in the Cabin - Web Scraping Puppy Data with Microsoft Fabric - 2nd Leg
32mModern Web Data Extraction: Tools, Techniques & Ethics
28mAutomate Your City Data With Python
52mUsing Python to Get Out the Vote
27mScratch, Dent and Data: Engineering A Used Appliance Information Advantage