Spark-driven Audience counting
summary
Describes an audience-counting pipeline that processes ~hundreds of thousands of events/sec and ~20 billion events/day into compact probabilistic sketches so per-campaign queries run in seconds. Uses Spark, Parquet/Hive and HyperLogLog-style sketches stored as ~1.5 KB blobs (tunable to ~4 KB for 1% error) to represent billions of device IDs and ~65,000 segments. Supports fast union, intersection and cardinality estimates across hours; typical queries return in ~2 seconds with ~2% error, avoiding huge DB indexes and cutting compute and storage costs.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 10:02Data scaling problem
- 23:31Apache Spark usage
- 31:27Query optimization
- 42:03Performance issues
- 53:16Data storage and archiving
related talks
29mWide Event Analytics
31mAnswering Multi-dimensional Queries on Massive Datasets
48mHyper Speed! When Big Data Blooms
48mLambda Architecture with Apache Spark
39mBuilding Next-Generation Audience Targeting and Analytics at NBCUniversal
28mBest of Stream + Batch: When Streaming Analytics Isn’t
47mSpeeding Event Processing to 1.6M/sec: Badoo’s Experience
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute