Apache Beam and Google Cloud DataFlow
summary
Claims that Apache Beam provides a unified model for batch and stream processing and that Google Cloud Dataflow runs Beam pipelines as a fully managed service. Explains windowing, watermark and windowing triggers, early/late firings and accumulating panes to handle out‑of‑order and late events. Notes the 2016 donation to Apache, current Java SDK and Python SDK (~80%) support, Java runners today, and demos a WordCount pipeline run locally and on Dataflow writing results to Cloud Storage.
AI-written summary · about
chapters (13)
- 0:00Introduction and welcome
- 2:06What is Apache Beam
- 3:09Big data processing at Google
- 9:37From MapReduce to Flume Java
- 11:47Batch vs streaming processing
- 13:14MillWheel and streaming systems
- 16:24Watermarks in streaming
- 18:50Dataflow unified model
- 21:40Dataflow example with windowing
- 28:47Dataflow to Apache Beam
- 31:56Demo of Apache Beam portability
- 36:13Apache Beam vision and future
- 38:18Closing and resources
related talks
24mApache beam and google cloud dataflow
21mApache Beam (incubating)
47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-dive
47mApache Beam - what do I gain?
26mBeam and dataflow: a perfect combination of parallel processing
52mApache Beam, a unified model for Big Data processing
10mDemocratizing the pipeline with Apache Beam and Dataflow
41mApache Beam: Unified Data Processing