Apache Beam - what do I gain?
summary
Apache Beam implements the Dataflow model so one pipeline describes both batch and streaming, with concrete demos sending simulated air‑quality events to Pub/Sub, applying fixed windows (10s) with an early firing at 5s and accumulating mode, then running the identical code on Apache Flink and Google Cloud Dataflow. The portability plan uses a portable runner + job server + SDK harness (Docker) to translate protobuf pipelines; measured SDK harness overhead is about 5–15%, and the demos work end‑to‑end.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:50Air quality data problem
- 12:41Beam programming model
- 29:01Portability and SDKs
- 35:06Demo on Apache Flink
- 38:19Portability framework
- 41:50Demo on Apache Spark
- 44:13Summary and resources
- 45:16Q&A
related talks
41mApache Beam: Unified Data Processing
21mApache Beam (incubating)
39mApache Beam and Google Cloud DataFlow
52mApache Beam, a unified model for Big Data processing
42mApache Beam: A Unified Model for...
29mApache Beam Making Big Data Portable With gRPC
26mBeam and dataflow: a perfect combination of parallel processing
24mApache beam and google cloud dataflow