From Flink 1.3 to 1.4
summary
Flink 1.3 adds concrete features: incremental checkpoints shrink checkpoint size (example production case 60 GB -> 1–30 GB) and cut checkpoint time (180s -> ~30s), while fine-grained recovery restarts only affected parts of the execution graph. API adds side outputs and table/CEP improvements; runtime work targets event-driven I/O and application-level flow control, a new deployment model with dynamic scaling, Kafka 0.11 connector for end-to-end exactly-once, optional Hadoop deps and child-first classloading, plus future plans for state replication, side inputs and programmatic job control.
AI-written summary · about
chapters (10)
- 0:00Introduction and 1.3 stats
- 2:45API improvements
- 5:09State handling and checkpoints
- 12:56Complex event processing
- 16:25Flink 1.4 network stack
- 20:47Deployment and scalability
- 22:31Kafka integration
- 23:40Operational improvements
- 26:35Future outlook and vision
- 33:49Q&A session
related talks
31mRethinking of fault tolerance in Flink: what lies ahead?
40mTowards Flink 2.0: Unified Batch & Stream Processing
20mFlinking, Fast and Slow
30mBlink: Improvements to Flink & its application in Alibaba Search
40mGrowing a managed Flink streaming platform from scratch
35mLessons learned on Apache Flink application availability in a hosted Flink service
16mZero-downtime upgrades of Flink applications
33mMaintaining highly available Flink applications at scale