Building Real-time Systems with Open Source Technologies
summary
Claims a working real-time analytics stack built with Kafka Storm Druid Hadoop that handles over 10 terabytes daily and trillions of events, where most queries return in milliseconds. The pipeline puts events into Kafka, processes joins and windows in Storm, writes real-time segments into Druid and uses Hadoop batch jobs to reprocess and replace older segments; it achieves second‑scale visibility but loses a few percent of late joins and requires batch reprocessing for exact corrections.
AI-written summary · about
chapters (5)
- 0:00Start
- 1:23THE PROBLEM Arbitrary and interactive exploration of time series data Online advertising System/application metrics
- 9:44DRUID Open sourced in Oct, 2012 Growing Community .45- contributors from many different organizations Designed for low latency ingestion and aggregation - Optimized for the types of queries we were trying to make
- 23:55WHAT WE GAINED Druid queries reflect new events within seconds Systems are fully decoupled Brief processing delays during maintenance . Because we need to restart Storm topologies But query performance and availability are not affected
- 28:20GET RADICAL Queries answered quickly, on fresh data Kafka provides fast, reliable event transport Storm and Hadoop clean and prepare data for Druid Druid handles queries and manages the serving layer • Real-time Analytics Data Stack
related talks
46mScalable Realtime Analytics using Druid
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid
37mBuilding an Event Analytics Pipeline with Kafka, ksqlDB, and Druid
40mBuild Real-time Analytic Applications: The Easy Way
20mAnalyzing streaming data with Apache Druid
20mReal-time analytics with Flink and Druid #bbuzz
27mSucceeding with Apache Druid and Clickstream Data
47mBuilding a Real-Time Analytics Database