Building Real-time Pipelines with FLaNK: A Case Study with Transit Data
summary
Shows an end-to-end pipeline that ingests GTFS realtime and camera feeds, enriches and joins them by lat/long, and exposes JSON materialized views for notebooks and apps. Uses FLaNK SQL to join Kafka topics, nii for ingestion and Python transforms, and pushes results to Iceberg/Kudu or REST endpoints. Demonstrates running on ~100 nodes, handling up to hundreds of millions of records/sec, and surviving a 100,000-record backlog while calling LLMs (Google Gemini) for summarization.
AI-written summary · about
chapters (6)
- 0:00Introduction to real-time pipelines
- 2:10Why use transit data
- 11:01Key technologies in FLaNK
- 18:54Demo: Transit data pipeline
- 25:18Demo: Additional data sources
- 30:57Q&A and closing
related talks
46mSmart Transit: Real-Time Transit Information with FLaNK
47mBuilding Modern Data Streaming Apps with Open Source
39mDeploying Fast Data Pipelines
41mAdding Generative AI to Real Time Streaming Pipelines
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
19mBuilding Stateful Streaming Pipelines
53mIntroduction to data streaming
32mStream Processing at Lyft with Flink and Beam