Automate Your City Data With Python
summary
Shows a concrete pipeline that turns municipal PDFs into a searchable DB using Python, PyMuPDF (fits) to rasterize pages, Tesseract OCR to extract text, sqlite-utils/Datasette to load and serve, and GitHub Actions to schedule builds. Demonstrates scraping Legistar-style sites (Pittsburgh example), handles recoverable folder-of-folders ETL, indexes full‑text search, and has run up to 57,000 pages for Berkeley and 1.3 million pages across civic.band. Runs on one Mac Mini and costs about $45/month.
AI-written summary · about
chapters (9)
- 0:00The importance of civic data
- 2:48Data origins and legality
- 6:03ETL process overview
- 7:27Extraction and API usage
- 14:37Transforming documents
- 17:39Loading data with Datasette
- 20:28Practical data analysis
- 22:29Deployment and maintenance
- 24:43Q&A and future outlook
related talks
18mAutomate Your City Data with Python
43mPython at City Scale
22mPython Applications in Infrastructure Planning and Civil Engineering
45mPython tools for webscraping
42mInside Out - Creating a Python Ecosystem for Analysis and Better Government
47mYou Got 10 Million PDFs from the Ministry. Now What?
28mData Harvest: Unlocking Insights with Python Web Scraping
19mDiofanti.org: A Civic Tech Journey Powered by PostgreSQL