Batching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
summary
Claims a migration from ETL batch to streaming lets analytics scale from 1 to 100+ million users in a month and handle about 1 billion page views per month with ~300–400 million sessions. The pipeline uses Apache Flink on Kinesis Data Analytics to produce one‑minute pre‑aggregates, writes parquet to S3 for Athena batch work, and serves dashboards from Elasticsearch/Kibana; MongoDB buffering is removed and real‑time A/B and alerting work in production.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:16Real-time tracking importance
- 3:24SpeedKit for e-commerce
- 8:01Tracking performance data
- 10:27Daily aggregates problems
- 12:55Pipeline rebuild for scalability
- 15:22Streaming with Flink & Kinesis
- 18:31Real-time A/B testing results
- 20:43Grafana RUM dashboard
related talks
27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
24mGive meaning to 100 billion analytics events a day, analytics at Teads
39mLarge scale streaming analytics
28mBest of Stream + Batch: When Streaming Analytics Isn’t
35mStreaming and Real-Time Analytics
23mCustomer Journey with streaming data on AWS
11mBig Data Analytics: Immutability & Idempotence
35mStreaming and Real-time Analytics