Webscraping Responsibly for Social Science Research
summary
Urban Institute data team describes concrete methods and outcomes: Python requests and Selenium with an add-on called site monitor throttle requests and measure latency; crawling DC Superior Court yields about 151,000 cases and an estimate of 68,000 people with records (≈34,000 without convictions); seventy‑five thousand LEHD gz files get merged via AWS and Spark into one dataset and indexed in Apache Solr; IRB screening, robots.txt checks, header contact info, night runs and PII stripping prevent operational and legal harm.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:27Web scraping power in social science
- 6:24Why scraping matters
- 9:09Example projects
- 16:12Legal risks of scraping
- 19:20Tools for responsible scraping
- 28:25Getting blocked consequences
- 30:10Scraping policy
- 35:07Summary
related talks
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations
32mModern Web Data Extraction: Tools, Techniques & Ethics
28mAutomate Your City Data With Python
42mScraping Your Way to a Dataset
14mHarnessing Common Crawl for AI and ML Applications
05m15-second Case Studies in Open Government Data
31mBeginning your own data engineering projects
45mData Driven Investigation in Defense of Human Rights