Location Based Data Engineering for Good with PySpark:Graphs
summary
Pipeline shows how PySpark plus Kedro reads cloud-stored pings, sessionizes by device, coalesces adjacent points with window functions, and infers trips using threshold rules, k‑means and longest‑dwell heuristics; an implemented flow outputs origin‑destination pairs and a graph of stops and neighborhoods that surfaces bottlenecks and modal mixes. Data quality uses Great Expectations and a Spark test library, plus versioning (dvc/dlt) and experimentation to quantify bias from home/work assumptions. Examples include parks, transit routes and play‑streets and the approach runs at scale.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:42What is Urban Logic
- 9:55Origin Destination Inference
- 11:15Initial Steps
- 14:20Look Ahead
- 19:00Bias
- 21:15Recap
- 21:40Data Engineering Tools
- 24:00Quality Engineering
- 25:27Observability
related talks
01h 36mSpark for Scala Developers
47mpython at scale – creating a High Definition Map of the world's road network
44mMake your data sexy
20mCloud Native Patterns for Geospatial Analytics and AI
24mData analytics with distributed tracing data
46mGeoSpatially enabling your Big Data Apache clusters with LocationTech
05mBig Data, No Big Deal: PySpark
15mFrom Air Quality to Aircraft and Automobiles, Data Is Everywhere