Improving Open Data Quality using Python
summary
Uses Great Expectations with pandas and sweetviz to validate Toronto open data bicycle thefts, not just talk about cleaning. Finds ~34k rows with ~28k missing OC_date (~83%) and ~6,700 non‑unique event IDs (~20%), plus inconsistent brand codes and undocumented extra files. Builds an expectation suite, validator and checkpoint, runs checks, documents results as local data docs, fixes many issues in pandas and reduces failing expectations (from four failing to two), while noting GE documents but does not auto‑fix data.
AI-written summary · about
chapters (13)
- 0:00Welcome
- 1:25Talk agenda
- 6:02Open data quality
- 17:50Data requirements
- 21:51Bicycle theft dataset
- 33:13Setting up the Python environment
- 36:02Exploratory analysis
- 54:24Great Expectations intro
- 1:02:59Validating data
- 1:11:50Fixing data with pandas
- 1:17:12Final validation
- 1:21:34Longitudinal data
- 1:24:28Q&A
related talks
17mOpen Source for Data Quality
25mForecasting critical food violations at restaurants using open data
35mBad data - anecdotes and examples from the real world
43mPython ❤️ Open Data!
42mTackling Homelessness with Open Data
35mDesigning for Impact in Civic Data Science
05mUsing OpenStreetMap with the Python OpenPoliceData package
33mData Science & Data Visualization in Python