Casting the spell: Druid in practice
summary
Nielsen Identity uses Apache Druid to serve subsecond, customer-facing analytics over tens of terabytes ingested daily and about 10 billion events per day. Cardinality relies on DataSketches KMV (default k=16k, bumped to 65k for accuracy), with known intersection bias mitigated by extracting selective dimensions (e.g., device type) and using lookups for slowly changing names. Batch ingestion uses adobe's ingestion with upstream Spark pre-aggregation and numShards tuning; changing the kill task interval shrinks deep storage from ~350 TB to 15 TB and cuts cost ~95%.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 2:40What you will learn?
- 3:38Business context
- 4:44Data infrastructure
- 5:33Building target audiences use-case
- 7:42Funnel analysis use-case
- 9:25The core challenge
- 10:19What is Theta Sketch?
- 13:06The Theta Sketch module in Druid
- 14:27High-level overview of the flow
- 15:29Data modeling
- 16:22Data model - introducing Theta Sketch
- 17:18Data model - mitigating Theta Sketch intersections
- 17:45Data model - slowly changing dimensions
- 19:35Data model - summary
- 20:19Why did we choose Hadoop-based ingestion?
- 21:15Hadoop-based ingestion tips
- 26:03Retention & deletion tips
- 26:52Retention & deletion example
- 27:50Dimension-based TTL
- 29:31Retention & deletion - summary
- 30:43Query optimization
- 32:56Queries - summary
- 33:27Bonus slide-read-only cluster
- 36:56Bonus slide - hardware
related talks
27mCasting the Spell: Druid in Practice
39mApache Druid 101
27mTales at Scale: Analytics at 1000 QPS and Beyond- 27mTales at Scale: Analytics at 1000 QPS and Beyond
20mAnalyzing streaming data with Apache Druid
45mHow We Got Druid to Work at Odnoklassniki
27mSucceeding with Apache Druid and Clickstream Data
28mBest of Stream + Batch: When Streaming Analytics Isn’t