Building an Apache Beam streaming runtime in GCP
summary
Google describes building a Beam/Dataflow runtime that scales globally: it implements trailing-edge watermarks for correct low-latency triggers, a distributed in-memory/hybrid shuffle service that spills to disk, and a separate state storage service so compute autoscaling adds workers without moving state. Real customers run hundreds of thousands of Dataflow jobs, shuffle jobs hit hundreds of terabytes, and measured shuffle times drop from ~1 hour to ~10 minutes and overall CPU efficiency improves.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:17History Lesson
- 2:58Lesson Learned: Watermarks
- 4:02What Triggers Output?
- 4:55Use Case: Anomaly Detection pipelines
- 7:44First attempt: leading-edge watermark
- 10:11Second attempt: dynamic leading edge watermark
- 11:08Trailing edge watermark
- 12:07Watermark: Definition
- 13:46Load Variance: Hand Tuning
- 19:11Techniques: Flow Control
- 25:22Techniques: Auto Scaling
- 28:44Load Variance: Lesson
- 31:15Spotify migrating the largest European Hadoop cluster to Dataflow
- 32:45Traditional Distributed Data Processing Architecture
- 33:47Shuffling key-value pairs
- 34:36Traditional Architecture Requires Manual Tuning
- 35:02Distributed in-memory Shuffle in batch Cloud Dataflow
- 36:59Faster Processing
- 37:33Supporting larger datasets
- 37:54What about streaming pipelines?
- 38:20Goal: Grouping by Event Time into Time Windows
- 44:07AB Tasty is using Dataflow Streaming Engine
- 45:05Main Takeaways
related talks
24mApache beam and google cloud dataflow
39mApache Beam and Google Cloud DataFlow
26mBeam and dataflow: a perfect combination of parallel processing
21mApache Beam (incubating)
47mApache Beam - what do I gain?
10mDemocratizing the pipeline with Apache Beam and Dataflow
56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...
41mApache Beam: Unified Data Processing