Storing State Forever: Why It Can Be Good For Your Analytics
summary
Shows a concrete Flink 1.12 implementation that keeps key state forever: a non-temporal union join that ingests tens of billions of CDC messages from Kafka into RocksDB, backfills to 13 TB of Flink state across 156 TaskManagers on GCP local SSDs, uses checkpoints to GCS, and hits 100% correctness versus the old lambda system. Describes RocksDB block cache issues, disabling generic type fallback to gain ~20% throughput, backfill time tradeoffs, and ~30-minute savepoint recovery observations.
AI-written summary · about
chapters (9)
- 0:00Introduction to Shopify
- 2:24Transitioning to streaming
- 4:07Designing streaming joins
- 6:01Handling late-arriving data
- 10:11Custom join implementation
- 13:52State management techniques
- 17:45Scalability and architecture
- 21:01Optimization and tuning
- 24:52Challenges and future outlook
related talks
32mState Unlocked
28mA look at Flink’s internal data structures & algorithms for efficient checkpointing
43mLarge Scale User Behavior Analytics by Flink
47mBring Flink to Large Scale Production
34mA journey of scaling high-volume low-latency stateful processing
28mMigration of a realtime stats product from Storm to Flink
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
38mApache Flink: A Very Quick Guide