Measuring 2.0 – Building the next generation webanalytics ...
summary
Retail site replaces JS-only analytics with a hybrid server+client pipeline that tags every page element, ships events via Kafka and writes session files to HDFS as Parquet/Avro, aiming for 50,000 measurements/sec and low-second latency. Processing uses Apache Flink for event-time windows, custom sessionizer and watermarks, with large window state spilled to HBase-backed visit state; anonymization salts long-term hashes. Practical issues: Kerberos token expirations, Flink exactly-once not guaranteed for Kafka output, and heavy work on schema distribution.
AI-written summary · about
chapters (11)
- 0:00Introduction to ball.com
- 2:19Measuring 2.0 motivation
- 5:13Design of data structure
- 7:54Hybrid measurement approach
- 10:39Sessionized lapa architecture
- 14:05Data processing and privacy
- 17:35System design and workflow
- 20:06Adopting Apache Flink
- 24:04Operator design and debugging
- 26:53Schema and storage strategies
- 30:28Q&A and closing remarks
related talks
27mArchitecting Sub-Second Analytics Pipelines
27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
41m"Analytics and how it should be!"
11mBig Data Analytics: Immutability & Idempotence
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
47mBuilding a Real-Time Analytics Database