Watermarks - Practical Tool for Any Streaming and Event Processing Application
summary
Watermarks give a concrete way to close event‑time windows: sources emit a watermark (min of per‑input maxima), processors advance when watermark passes a window boundary and then emit results. The demo builds an experimental streaming runtime on Spring Cloud Stream (compared to Apache Flink/Beam implementations) and works end‑to‑end. It uses 1‑minute windows and “3 attempts = fraud” rule, shows bounded disorder = 5 time units to tolerate out‑of‑order events, and adds idle‑input timeout to avoid stuck watermarks.
AI-written summary · about
chapters (13)
- 0:00Streaming watermarks intro
- 2:42Data intensive systems
- 6:15Stream processing basics
- 10:10Time domains and completeness
- 13:03Watermarks for streaming
- 17:59Generating watermarks
- 22:38Late and out-of-order data
- 25:26Propagating watermarks
- 31:24Watermark drawbacks
- 33:50Conformant vs non-conformant
- 35:56Key watermark concepts
- 37:00Implementing watermarks
- 38:48Live demo: watermark dynamics
related talks
29mWhen your stream doesn't stream very well
28mTimeless Streams
19mLarge Scale Real Time Ad Invalid Traffic Detection with Flink
35mKeynote: Stream analytics made real with Pravega and Apache Flink
43mBuilding a community fountain around your data stream
41mLevel Up Your Java Streams with Gatherers
32mProcess Data Streams Efficiently in Elixir
29mA brief history of time with Apache Flink