A look at Flink’s internal data structures & algorithms for efficient checkpointing
summary
Flink implements incremental checkpoints that record only deltas and asynchronous checkpoints that let the pipeline keep running; those features hinge on log-structured merge trees and copy-on-write hash tables assembled into a backend that uploads compacted SSTables with reference counting and per-entry version counters. The result cuts checkpoint size and duration far below full-state copies, supports frequent consistent checkpoints, and operates in production at cluster scale while allowing fast recovery from the retained checkpoint history.
AI-written summary · about
chapters (17)
- 0:00Introduction to Flink checkpointing
- 1:05Barrier snapshot algorithm explained
- 2:12Overhead of full state snapshots
- 3:54Incremental checkpointing benefits
- 5:39State backend and memory management
- 7:10Key-value store for checkpoints
- 9:03SSTable and compaction process
- 11:06Checkpoint directory structure
- 13:38Merging SSTables during checkpoints
- 15:00Handling user-defined state
- 17:32Snapshot versioning and tracking
- 19:03Copy-on-write for state updates
- 20:48Copy-on-write implementation details
- 22:36Garbage collection of old states
- 24:42Managing checkpoints in Flink
- 26:42Performance considerations
- 28:11Q&A and closing remarks
related talks
28mRundown of Flink's Checkpoints
31mRethinking of fault tolerance in Flink: what lies ahead?
36mApache Flink in the Cloud-native Era
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
36mFlink SQL: Unified Streaming and batch for hive integration
32mStoring State Forever: Why It Can Be Good For Your Analytics
38mApache Flink: A Very Quick Guide
42mApache Flink Worst Practices