Tale of Stateful Stream to Stream Processing
summary
Netflix migrates its end-to-end batch payment workflow for 180M+ members across 190 countries to a multi-step Apache Flink streaming pipeline, rewriting big batch jobs into smaller stateful stream-to-stream jobs that hold pending state up to 30 days. The engineering work uses Flink state and checkpoints, bootstrap and recovery logic, and scaled checkpoints to survive spikes, splitting consumers and joins into independent stages; results show zero data loss and production-grade accuracy while reducing end-to-end latency.
AI-written summary · about
chapters (5)
- 0:00Introduction to stream processing
- 15:09State management challenges
- 21:10Monitoring and metrics
- 26:20Applications and examples
- 30:35Q&A and closing
related talks
26mHow Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date
37mBuilding and operating large-scale streaming applications at King
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
32mBig Data Processing : Where do we go next ?
43mStream Processing and Applications in the Modern Age
42mApache Flink deep-dive #bbuzz
42mApache Flink Worst Practices
33mFlink and Beam: Current State & Roadmap