Predicting Hacker News with Beam and TensorFlow

on YouTube ↗21m

" summary

Demonstrates using Apache Beam and Google Cloud Dataflow with Java/Python SDKs to solve stragglers and cost by dynamic work rebalancing (aka liquid sharding), plus cluster autoscaling and in-place streaming updates. Beam adds splittable sources so runners (Flink, Spark, Dataflow) can split backlog, scale from ~10 to hundreds of workers, and reassign slow work without double-processing. Prototypes run in production and Dataflow spins up ephemeral clusters and tears them down automatically.

AI-written summary · about

" chapters (10)

1. 0:00Introduction and apologies
2. 0:35Evolution of distributed data processing
3. 2:46Motivation for Dataflow and Beam
4. 3:50Challenges in streaming pipelines
5. 6:42Handling straggling workers
6. 9:55Scaling workers in the cloud
7. 13:28Dataflow and Beam summary
8. 14:52Open source Beam SDK
9. 15:14Beam SDK demo
10. 17:25Q&A session

" related talks

• 10mDemocratizing the pipeline with Apache Beam and DataflowTyler Akidau, Reuven Lax · Strata Data Conference 2017 - New York, New York
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-diveSergei Sokolenko · Highload fwdays'19
• 43mNo shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou · Flink Forward Berlin 2016
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 52mApache Beam, a unified model for Big Data processingJean-Baptiste Onofré, Ismaël Mejía · DevFest Nantes 2016