Automate Your City Data with Python
summary
Pipeline description: Python scripts hit the city/primov APIs, iterate template IDs and download meeting PDFs into a folder of folders of PDFs, then split pages and run OCR (Tesseract) so text is ingestible. The pipeline follows extract transform load and inserts pages into a Datasette database with enable FTS true for full‑text queries. It handles tens of thousands (approaching 100k) pages, runs on Github actions or local hosts to avoid high AWS costs, and yields a working, queriable City minutes database.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:48Who is Philip James
- 4:14Goal: normalizing civic data
- 5:39Extracting civic website data
- 8:49Finding APIs with dev tools
- 11:18Downloading PDFs with Python
- 12:20Transforming PDFs to text
- 14:06Loading data into database
- 15:07Analyzing meeting data
- 16:08Deploying the database
- 16:52Future directions
related talks
28mAutomate Your City Data With Python
22mPython Applications in Infrastructure Planning and Civil Engineering
43mPython at City Scale
42mTackling Homelessness with Open Data
40mOpen Data Dashboards & Python Web Scraping
06mUse Your Data Skills for Good
45mPython tools for webscraping
42mInside Out - Creating a Python Ecosystem for Analysis and Better Government