Beam and dataflow: a perfect combination of parallel processing

on YouTube ↗26m

" summary

Explains how Apache Beam encodes logic as PCollections and Transforms to express batch and streaming pipelines, then deploys the same pipeline on Google Cloud Dataflow to show an auto-scalable managed runner that handles worker allocation, failures, and parallelism. Compares running locally, on Spark and Flink versus Dataflow, walks through a live demo building a Beam pipeline, submitting it to Dataflow, and verifying that scaling and resource management work end to end.

AI-written summary · about

" chapters (5)

1. 0:00Introduction to talk
2. 5:40Parallel processing concepts
3. 12:15Dataflow and Beam basics
4. 17:39Google Cloud integration
5. 21:06Demo and summary

" related talks

• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 10mDemocratizing the pipeline with Apache Beam and DataflowTyler Akidau, Reuven Lax · Strata Data Conference 2017 - New York, New York
• 52mApache Beam, a unified model for Big Data processingJean-Baptiste Onofré, Ismaël Mejía · DevFest Nantes 2016
• 41mApache Beam: Unified Data ProcessingAlexey Diomin · Voxxed Days Minsk 2018
• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018