Scalable Realtime Analytics using Druid
summary
Druid is presented as a columnar distributed store that delivers real-time ingestion with subsecond interactive queries by combining memory row stores on realtime nodes, columnar compressed segments on historical nodes, brokers for fan‑in, and deep storage (S3/HDFS). Demonstrations ingest Kafka streams (EC2 spot prices, air sensors), produce summarized segments and bitmap indexes, and show aggregation on realtime plus historical merges. Reported scale: about 500k events/sec average (2M peak), 50 PB raw across a production cluster, median query ~500 ms, 90th <1s and 95th <5s.
AI-written summary · about
chapters (10)
- 0:00Introduction to scalable realtime analytics
- 2:44Druid’s history and motivation
- 8:41What is Druid and key features
- 10:04EC2 price analytics demo
- 18:41Air quality sensor data analysis
- 22:43Druid architecture and node types
- 27:13How Druid stores and processes data
- 33:30Production cluster performance and scale
- 37:01When Druid isn’t suitable
- 38:24Q&A
related talks
45mDruid: Powering Interactive Data Applications at Scale
20mAnalyzing streaming data with Apache Druid
39mApache Druid 101
40mBuild Real-time Analytic Applications: The Easy Way
32mBuilding Real-time Systems with Open Source Technologies
20mReal-time analytics with Flink and Druid #bbuzz
27mSucceeding with Apache Druid and Clickstream Data
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid