Smart Transit: Real-Time Transit Information with FLaNK
summary
A NiFi pipeline ingests geoRSS and RSS feeds, unzips or parses XML into JSON, normalizes a tab‑delimited point into latitude/longitude, and adds a unique id and timestamp before publishing to a streaming layer (demo uses Apache Pulsar and can target Apache Kafka). Flink runs SQL over the topics to join weather, transit and camera feeds; events land in Apache Kudu for Impala/Hue queries and trigger Slack alerts and a microservice API. Demo processes thousands/sec (throttled for visibility) and the end‑to‑end flow works.
AI-written summary · about
chapters (9)
- 0:00Introduction to Smart Transit
- 4:02Introducing the FLiNK stack
- 10:23Processing GTFS transit data
- 16:44Demo: Real-time transit data ingestion
- 24:07Demo: Transit data to Pulsar and Flink
- 33:37Merging transit data streams
- 38:14Weather and transit data integration
- 40:42Custom processors in Apache NiFi
- 45:13Closing and future talks
related talks
31mBuilding Real-time Pipelines with FLaNK: A Case Study with Transit Data
47mBuilding Modern Data Streaming Apps with Open Source
36mHail Hydrate! From Stream to Lake
39mCodeless Pipelines With Pulsar and Flink
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
43mA Data Streaming Architecture with Apache Flink #bbuzz
43mContinuous SQL with Kafka and Flink
47mSpeeding Up the City with Apache Flink