When ordering matters
summary
Bol.com measures at the page-build point and routes events into Kafka partitions so downstream Flink can do stateful per-session processing with low latency. The pipeline handles roughly 100k events/sec and ~3 TB/day, uses exactly once and per-session ordering guarantees, and serializes with a compact Avro messageformat that embeds a 64-bit fingerprint schema for evolution. Behavioral analytics run as finite-state (or pushdown) automata to detect cause–effect (click → buy) and power live features like streaming search suggestions instead of nightly batch updates.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 3:59Measuring interactions accurately
- 9:07Real-time processing needs
- 13:00Why ordering matters
- 16:57End-to-end ordering guarantees
- 21:15Stateful processing systems
- 24:50Schema evolution with Avro
- 28:25Streaming search suggestions demo
- 32:42Closing and Q&A
related talks
29mStreaming & Eventing for Top Management
38mBuilding Unified Streaming Platform at Uber
34mCustom, Complex Windows at Scale using Apache Flink
25mMixing data streams at scale
43mStructure and Interpretation of Stream Processing
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
45mEvent-based system: Uklon practice
01h 04mState or Events? Which Shall I Keep?