The Dask at Hand: Using Dask to Speed up the High-Quality Transit Areas dataset
summary
Builds a reproducible pipeline that derives California’s legally defined high quality transit corridors from GTFS and publishes them on the CA open data portal monthly, using Python, Dask (Dask DataFrames), geopandas and shapely with BigQuery and JupyterHub integration. Iterations cut runtime from 10 hours to half that with Dask dataframes then to under a minute by eliminating unnecessary loops, introducing direction-aware spatial joins and narrowing to ~100k route pairs before rowwise geopandas intersections; the dataset is produced monthly and available publicly.
AI-written summary · about
chapters (7)
- 0:00Dask for transit data
- 2:56Why Dask for geospatial processing
- 5:48Legislation into code
- 9:25Intersecting transit routes
- 13:44Code optimization results
- 16:12Lessons from Dask
- 18:41Conclusion
related talks
01h 26mDask tutorial
44mDask for ad hoc distributed computing
01h 23mData Science at Scale with Dask
01h 03mDask: Next Steps in Parallel Python
01h 17mScaling PyData with Dask
32mEnabling real-time insights through stream processing in Python
10mCivic Hacking & Urban Informatics: Mapping Data with GeoPandas
40mData-centric exploration using intake, dask, hvplot, and more