Real-time analytics with Flink and Druid #bbuzz
summary
A streaming-only pipeline connects Kafka -> Flink -> Druid to deliver sub-second multidimensional queries at 70,000 events/sec with records carrying up to 50 dimensions and 20 metrics; Flink provides exactly-once processing and a single code path for live and replayed data, Druid stores hourly segments partitioned by time, and a contributed Flink sink to Tranquility plus custom replay logic from HDFS lets the team reprocess history and run real-time reporting successfully.
AI-written summary · about
chapters (9)
- 0:00Introduction to realtime data
- 2:46Introduction to Apache Druid
- 4:25Druid storage optimization
- 5:59Data ingestion and Flink
- 7:16Tranquility and Flink sync
- 9:08Lambda vs Kappa architecture
- 11:04Implementation challenges
- 13:54Conclusion and results
- 14:50Q&A session
related talks
46mScalable Realtime Analytics using Druid
20mAnalyzing streaming data with Apache Druid
27mSucceeding with Apache Druid and Clickstream Data
37mBuilding an Event Analytics Pipeline with Kafka, ksqlDB, and Druid
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid
32mBuilding Real-time Systems with Open Source Technologies
45mDruid: Powering Interactive Data Applications at Scale
39mApache Druid 101