No shard left behind: Dynamic work rebalancing in Apache Beam

on YouTube ↗43m

" summary

Apache Beam implements dynamic work rebalancing in Google Cloud Dataflow by splitting running tasks and reassigning chunks to idle workers, reducing idle time in a 400 VM job and reportedly saving about 55 minutes on a roughly 20-minute baseline run. Runners call APIs like split_at_fraction, get_fraction_consumed and get_backlog_bytes to decide when to split; this enables effective autoscaling (observed scaling from tens to ~1,000 workers) and works in live demos.

AI-written summary · about

" chapters (14)

1. 0:00Introduction and welcome
2. 0:26Apache Beam overview
3. 5:10Pipeline variability in production
4. 7:25The curse of stragglers
5. 11:46Dynamic adaptation philosophy
6. 13:14Bundle flexibility in API
7. 15:44Source API and splitting
8. 18:13Solving stragglers with rebalancing
9. 21:03Dynamic work rebalancing in practice
10. 22:53Link to auto scaling
11. 25:01API for progress tracking
12. 27:37Parallelism limits and scaling
13. 29:19Apache Beam's unified model
14. 30:45Q&A session

" related talks

• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 21mPredicting Hacker News with Beam and TensorFlowEric Anderson · Data By the Bay 2016
• 04mDBTB INT Eugene Kirpichov rEugene Kirpichov · Data By the Bay 2016
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...Bartosz Wieczorek · GeeCON 2023
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017