Casting the spell: Druid in practice
summary
Uses Apache Druid with DataSketches to serve real-time dashboards on tens of terabytes ingested daily, relying on Theta sketch (KMV) for approximate distinct counts and set operations. Pipeline writes pre-aggregated sketches from Spark, ingests via Hadoop tasks, and tunes sketch size (4k vs 65k, default 16k) to trade accuracy for speed. Manages retention with load/drop pools and kill tasks — changing kill interval cuts deep storage from ~350 TB ($8k/mo) to 15 TB ($350/mo), a 95% storage reduction, and Druid SQL plus vectorized engine speeds sketch queries.
AI-written summary · about
chapters (8)
- 0:00Introduction to Druid
- 2:46Business context at Nielsen
- 6:16Why Druid for big data
- 10:50Data modeling best practices
- 15:20Batch ingestion setup
- 19:12Retention and deletion strategies
- 23:19Query optimization tips
- 26:27Summary and resources
related talks
27mCasting the Spell: Druid in Practice
45mDruid: Powering Interactive Data Applications at Scale
46mScalable Realtime Analytics using Druid
39mApache Druid 101
40mBuild Real-time Analytic Applications: The Easy Way
27mSucceeding with Apache Druid and Clickstream Data
45mHow We Got Druid to Work at Odnoklassniki
31mAnswering Multi-dimensional Queries on Massive Datasets