Apache Beam: A Unified Model for...
summary
A programming model that unifies batch and streaming, expressing pipelines (example: WordCount and per-user hourly sums written to BigQuery) and running them on multiple backends including Flink, Spark, Google Cloud Dataflow; it implements event-time windowing with timestamps, watermarks that estimate completeness, and composable triggers such as after watermark past end of window and early/late firings (example: early firing every minute, count-based firing after 2 elements) and supports accumulation vs discarding modes; project is incubating and the multi-runner demos execute successfully.
AI-written summary · about
chapters (8)
- 0:00Introduction to Apache Beam
- 1:07Big Data processing challenges
- 8:26Apache Beam programming model
- 12:58Computing in Beam pipelines
- 15:47Event time vs processing time
- 21:02Windowing and triggers
- 31:37Apache Beam project overview
- 35:29Q&A session
related talks
21mApache Beam (incubating)
41mApache Beam: Unified Data Processing
52mApache Beam, a unified model for Big Data processing
47mApache Beam - what do I gain?
39mApache Beam and Google Cloud DataFlow
46mBuilding an Apache Beam streaming runtime in GCP
16mYour Data Pipeline with Apache Beam
26mBeam and dataflow: a perfect combination of parallel processing