Autoscaling Apache Flink Applications

on YouTube ↗20m

" summary

Three scaling modes are compared: manual savepoint-based rescaling (stop, take savepoint, restore), Flink 1.13 reactive scaling which auto-expands parallelism when nodes appear (supported on standalone, Kubernetes, Docker), and upcoming autoscaling where Flink itself requests capacity. Reactive demo drives CPU with a sine waveform and moves taskmanagers 1→5 using Kubernetes HPA on CPU, but surprise scale-downs cause extra reprocessing. Implementation adds declarative resource management and a new scheduler to compute new distributed execution graphs and request exact resources.

AI-written summary · about

" chapters (6)

1. 0:00Intro
2. 1:22Apache Flink
3. 3:19Why Scale
4. 5:00How to Scale
5. 13:22Implementation
6. 15:49Questions

" related talks

• 43mDynamic Scaling - How Apache Flink adapts to changing workloadsTill Rohrmann · Flink Forward Berlin 2016
• 38mAutoscaling Flink at NetflixTimothy Farkas · Flink Forward Virtual 2020
• 30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?David Moravek · Flink Forward Global 2021
• 44mNews from Flink’s engine room: “Full steam ahead”Till Rohrmann · Flink Forward Global Virtual 2020
• 37mIntegrate Flink with Kubernetes nativelyYang Wang · Flink Forward Virtual 2020
• 42mManaging Apache Flink on Kubernetes - FlinkK8sOperatorAnand Swaminathan · KubeCon + CloudNativeCon North America 2019
• 36mApache Flink in the Cloud-native EraStephan Ewen · Flink Forward Global 2021
• 37mStateful Stream Processing with Apache Flink 1.5 and beyondNico Kruber · Berlin Buzzwords 2018