Maintaining highly available Flink applications at scale
summary
KDA implements a standby Flink cluster that is provisioned before stopping the live job, reusing the most recent checkpoints over savepoints and locking application state so checkpoints remain compatible. Automation adds a new maintenance state and configurable maintenance windows. Result: end to end downtime drops from ~120+ seconds to about 10–30 seconds for most apps, a reported 4x–12x improvement, with higher resource use during transient failovers and automated, auditable rollouts that already run in production.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:39Amazon Kinesis Data Analytics (KDA)
- 2:29Building with Amazon Kinesis Data Analytics
- 3:23How do we run Apache Flink?
- 4:38Behind the scenes . On a Start Application call
- 6:33Problem
- 11:33What is End to End Latency
- 13:45Stateful Failover - Sequential
- 14:48Stateful Failover - Standby Flink Cluster
- 18:03What state to use for failover?
- 20:07State capture modes
- 22:08Customer visibility and configurability
- 26:05Optimizations recap
- 28:29Next - Towards zero end to end latency
related talks
35mLessons learned on Apache Flink application availability in a hosted Flink service
16mZero-downtime upgrades of Flink applications
31mRethinking of fault tolerance in Flink: what lies ahead?
38mFrom Flink 1.3 to 1.4
37mBuilding and operating large-scale streaming applications at King
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
58mBuilding Self Service Platforms for Apache Flink
40mGrowing a managed Flink streaming platform from scratch