Apache Flink in the Cloud-native Era

on YouTube ↗36m

" summary

Apache Flink is moving toward cloud-native operations with concrete features: buffer debloating reduces in-flight data so checkpoints become snappier, aiming for predictable 2–5 second checkpoint cadence; generalized log-based incremental checkpoints decouple checkpointing from heavy compactions to avoid long snapshot delays; reactive scaling mode lets jobs scale when Kubernetes adds or removes pods; a demo shows persistent volumes for failover recovery cutting state restore from ~10 seconds to under 1 second.

AI-written summary · about

" chapters (8)

1. 0:00Apache Flink's stats in the ASF 2020
2. 3:14Apache Flink in the Cloud-native Era
3. 11:25Four initiatives that change Flink's Cloud-native Nature
4. 11:38Better Operations APIs
5. 12:41Predictable Fast Checkpoint Times
6. 19:08Elasticity
7. 25:41Faster Failover
8. 29:30Demo - Persistent Volumes for Failover Recovery

" related talks

• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 31mRethinking of fault tolerance in Flink: what lies ahead?Yuan Mei, Piotr Nowojski · Flink Forward Virtual 2020
• 28mA look at Flink’s internal data structures & algorithms for efficient checkpointingStefan Richter · Flink Forward Berlin 2017
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 26mRunning Apache Flink on AWSRyan Nienhuis · Flink Forward Europe 2019