Custom, Complex Windows at Scale using Apache Flink
summary
Netflix moves batch badge ETL into Apache Flink by building a custom Window API layer that creates unaligned, event‑type bounded windows, handles millions of events per second (hundreds of billions per day, tens of GB/s ingress), copes with out‑of‑order and late events, emits periodic early results, and trims retained data so only session‑relevant events persist. Implementation stores state in a RocksDB backend (resulting in tens of terabytes of state) and uses merge/trigger logic; process functions are noted as a simpler alternative.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:43Agenda
- 1:28Use Cases
- 5:07Requirements
- 9:20Solution
- 9:40Goal State
- 12:41Event Processing Flow
- 14:25Window API
- 17:37Code
- 18:08Merge windows
- 19:35Memory leak
- 20:31Windows as key
- 20:55Serializer
- 21:23Serialization Code
- 21:40Object Free Use
- 22:12Immutable
- 22:39End Event
- 23:09Calculate Session Windows
- 23:23Fire Session Windows
- 23:35Exploratory Windows
- 24:17Merge Window
- 25:25Summa flanks
- 25:56Event time timer
- 26:11Early firings
- 26:35Event time
- 27:01Update early firing
- 27:17Pitfalls
- 29:04Looking Back
- 29:57Process Function
related talks
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
29mStreaming & Eventing for Top Management
36mWhen ordering matters
38mBuilding Unified Streaming Platform at Uber
37mKeynote | Flink First: Designing Modern Trading Systems
37mNot So Big – Flink as a true Application Framework
43mStructure and Interpretation of Stream Processing
32mFlink's SQL Engine: Let's open the engine room!