Robust Stream Processing with Apache Flink
summary
Live demo builds a Flink job that generates sawtooth/sine/square sensor streams, writes to InfluxDB and Grafana, windows by key and sums values, and shows concrete behaviors: with processing time the sine sum drifts, with event time fixes non-deterministic sums the per-window sum is exactly zero; disabling checkpoints causes a multi-minute data loss when a TaskManager dies, enabling checkpoints yields instant consistent recovery, and savepoints resume across clusters so the job restarts from an exact prior state even after cluster destruction or code changes.
AI-written summary · about
chapters (38)
- 0:00Introduction
- 1:30Stream Processing
- 2:01Stateful
- 3:07Graphs
- 4:34Input Source
- 5:05Square Wave
- 6:31Flink Cluster
- 6:51Flux DB Sync
- 8:19Win to it
- 9:50Pressure wave example
- 12:07Windowing the data
- 12:42Setting the time characteristic
- 14:14Failure Handling
- 15:44Data Flow Graph
- 17:30Socket Stream
- 18:10Amplifier Function
- 20:02Dynamic Output
- 20:19Failures
- 20:40No Data
- 20:51Resource Recovery
- 21:10Flink UI
- 21:40Troubleshooting
- 21:51Looking back
- 22:23Flink checkpoint
- 22:47Stateful operators
- 23:26Reprogramming
- 25:04Fail
- 25:31Data Gap
- 25:41Testing
- 26:03Full Recovery
- 26:23Downtown Handling
- 26:55Save Points
- 27:37Destroy Cluster
- 27:55Create New Cluster
- 28:18Offloading Cluster
- 29:31Code Changes
- 30:11Save Point
- 31:04Save Points and Checkpoints
related talks
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
43mA Data Streaming Architecture with Apache Flink #bbuzz
38mApache Flink: A Very Quick Guide
46mFlink SQL in 2020: Time to show off!
45mApache Flink — Real-Time Data Processing
27mIntroduction to Flink in 30 minutes
41mBeyond the Hype: When Apache Flink Solves Real Problems