Enhancing Civics Data with Elasticsearch, pandas and Eland
summary
Project consolidates San Diego "Calls for Service" across 11 separate datasets dating back to 2015, cleaning CSVs with pandas and helper scripts to standardize types and drop duplicates. It loads a single-year sample of 48,976 records (nearly 4 million records total historically), converts approximate locations to geo_point via public geoJSON and Elasticsearch filters, bulk-ingests and syncs data through eland-backed DataFrames into Elasticsearch for Kibana visualization, and automates periodic updates — full pipeline runs and dashboards function as demonstrated.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:57Disclaimer
- 7:21Downloading Data
- 8:40Importing pandas
- 9:38Loading police call data
- 10:20Data frame binding
- 10:51Merging data
- 13:13Sorting data
- 17:14Selective enforcement
- 18:37Uploading Data
- 19:19Loading Data into Elasticsearch
- 27:05Conclusion
related talks
39mHandling, extending and presenting data with ELK+
35mEnriching Postal Addresses With Elastic Stack
41mBetter Data Exploration and Visualisation with Elasticsearch and Kibana
44mManagement of Data Communities
35mVisualising Public Service: Seeing Inside Birmingham City
10mGetting Your Data into Elasticsearch: Let Me Count the Ways
28mAutomate Your City Data With Python
40mWebscraping Responsibly for Social Science Research