Succeeding with Apache Druid and Clickstream Data

on YouTube ↗27m

" summary

Explains how Apache Druid scales ingestion to millions of events per second and serves high-concurrency interactive analytics, using columnar compression, time-based partitions and sub-partitioning to prune queries. Describes real pipelines built with Kafka and Kinesis, streaming via Flink or Spark, plus compaction for late data. Recommends upstream session assembly and roll-up, and uses HyperLogLog and Theta sketches for fast distinct and set approximations; introduces a delta-of-event state pattern for near-real-time counters.

AI-written summary · about

" chapters (3)

1. 0:005:49 Introduction to Apache Druid & Imply
2. 5:5016:50 Clickstream Analytics & Pipelines
3. 16:5124:25 Critical Techniques

" related talks

• 28mBest of Stream + Batch: When Streaming Analytics Isn’tRohan Garg · DataEngBytes 2022
• 20mAnalyzing streaming data with Apache DruidDarin Briskman · AWS re:Invent 2023
• 39mApache Druid 101Matt Sarrel · Data Con LA 2020
• 41mBuilding an Open Source Streaming Analytics Stack with Kafka and DruidFangjin Yang · Embedded Linux Conference + OpenIoT Summit North America 2017
• 46mScalable Realtime Analytics using DruidNishant Bangarwa · The Fifth Elephant 2016
• 37mBuilding an Event Analytics Pipeline with Kafka, ksqlDB, and DruidHellmar Becker · Big Data Conference Europe 2023
• 20mReal-time analytics with Flink and Druid #bbuzzJan Graßegger · Berlin Buzzwords 2016
• 45mDruid: Powering Interactive Data Applications at ScaleFangjin Yang · Strange Loop 2016