Casting the Spell: Druid in Practice
summary
Apache Druid powers a pipeline that ingests tens of terabytes daily (10 billion events/day, >5 PB lake, 60 TB/day on S3) using Hadoop-based ingestion with numShards and affinity to scale; distinct counts use DataSketches (KMV), tuning sketch size (default 16k, examples 4k vs 65k) to trade accuracy for latency. Lookups replace mutable audience names to avoid reindexing. Retention uses load/drop rules plus kill tasks; widening kill task intervals removes unused segment versions and cuts one S3 bill by ~95% for a large data source.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 2:56Business context
- 3:24Data infrastructure
- 3:59Building target audiences use-case
- 5:17Funnel analysis use-case
- 6:34The core challenge
- 7:13What is Theta Sketch?
- 9:12The Theta Sketch module in Druid
- 10:20High-level overview of the flow
- 10:58Data modeling
- 11:39Data model - introducing Theta Sketch
- 12:50Data model - slowly changing dimensions
- 13:14Data model - lookups
- 14:25Data model - summary
- 15:42Hadoop-based ingestion tips • Parallel execution of ingestion tasks from seperate Hadoop clusters
- 17:09Ingestion - summary
- 18:06Retention & deletion tips
- 18:53Retention & deletion example
- 19:31Dimension-based TTL
- 21:03Retention & deletion - summary
- 21:38Why did we choose native queries?
- 22:00Query optimization
- 24:17Bonus slide-read-only cluster
related talks
28mCasting the spell: Druid in practice
42mCasting the spell: Druid in practice
45mDruid: Powering Interactive Data Applications at Scale
39mApache Druid 101
27mSucceeding with Apache Druid and Clickstream Data
46mScalable Realtime Analytics using Druid
20mAnalyzing streaming data with Apache Druid
27mTales at Scale: Analytics at 1000 QPS and Beyond