Advances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-dive
summary
Clear claims and numbers: Apache Beam + Google Cloud Dataflow use a backend shuffle and separate state storage from compute so large-grouping jobs drop from hundreds of workers to tens and show up to 5x faster on real pipelines. A Dataflow Streaming Engine stores window state centrally, improves autoscaling, and immediately scales down after load. SQL extensions (tumbling windows) let a Pub/Sub stream join a BigQuery lookup and emit rolling results every 5 seconds with exactly-once processing for near‑real‑time dashboards.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:24Overview of Google Cloud
- 2:11Google Cloud Services
- 5:39MapReduce
- 6:15Floom
- 9:28Millwheel
- 10:06Practical things
- 12:45Beam and Dataflow
- 14:16Beam portability
- 15:42Lessons learned
- 16:04Traditional data processing architectures
- 17:11What is shuffling
- 18:13Bringing your network down
- 18:33Shuffle Dataflow
- 19:59Results
- 21:01Steaming
- 22:14Dataflow steaming engine
- 23:43Dataflow steaming engine comparison
- 27:26Summary
- 27:40Streaming is hard
- 28:12Making steaming less difficult
- 28:53Dataflow sequel in BigQuery
- 29:21Demo
- 36:33Recap
- 37:04Audience questions
related talks
39mApache Beam and Google Cloud DataFlow
52mApache Beam, a unified model for Big Data processing
10mDemocratizing the pipeline with Apache Beam and Dataflow
21mPredicting Hacker News with Beam and TensorFlow
41mReal-time Data Processing with Cloud Dataflow
25mData: Moving with the flow
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
39mStreaming doesn’t have to be hard