Enabling real-time insights through stream processing in Python
summary
Shows actual tooling and results: uses CSP (C++ core with Python API), Bitewax, Flink and Kafka comparisons, GTFS/GBFS feeds and Parquet playback. Builds an interactive dashboard that computes average wait time ~4.5 min plus 90th/99th percentiles from MTA GTFS (30s updates), and bike availability from Citi Bike: 2200 stations 29,000 bikes with 99 empty 119 full (60s updates). Demonstrates historical playback via Parquet and seamless switch to real‑time execution; dashboard updates live and the demo works end to end.
AI-written summary · about
chapters (15)
- 0:00Real-time stream processing intro
- 1:04What we'll build
- 3:10Human streaming data
- 4:32Streaming challenges
- 8:23Python stream tools
- 11:50Composable stream processing
- 13:15Core CSP concepts
- 17:04Modular stream graphs
- 19:09Industries using streams
- 19:29Demo setup NYC transit
- 25:54Subway wait times
- 26:57Transit alerts handling
- 27:38Bike share metrics
- 29:47Live transit visualization
- 31:55Key takeaways
related talks
27mReal-time Event Streaming with Python
40mPython and pandas as back end to real-time data driven applications
39mStraightforward stream processing with CSP
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
20mStreaming DataFrames: A New Way to Process Streaming Data
01h 49mBuilding Web-based Analysis & Simulation Platforms
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
01h 12mGPU-Accelerated Data Analytics in Python