Apache Beam: Unified Data Processing
summary
Apache Beam provides a single programming model that runs on Dataflow, Flink, Spark, Apex and vendor runners and exposes portable runner and job/server architecture so a pipeline can mix runners and languages (Java, Python, Go). It implements keyed state and timers with backends like RocksDB state backend, supports session/fixed windows, joins and a new splitable DoFn for dynamic parallelism. At production scale teams report half‑billion events/sec and 125k transactions/sec; current portable RPC prototype adds about ~15% overhead.
AI-written summary · about
chapters (30)
- 0:00Apache Beam introduction
- 1:17Apache Beam history
- 2:25Data model and runners
- 3:51Vendor and community support
- 5:22Beam vs Spark
- 6:05Supported programming languages
- 7:10Dataflow SDK and Go
- 8:16Production stability
- 8:37Apache Beam basics
- 10:25PCollection and PTransform
- 11:54Standard transformations
- 12:57State handling
- 16:11State and window management
- 17:58State issues and timers
- 19:04Timers and usage
- 21:15State in runners
- 23:27Current state and outlook
- 24:07Split dofn limits
- 25:53Source interfaces evolution
- 26:57SQL and window support
- 28:46SQL in Apache Beam
- 30:11Code generation
- 30:59Portable runner architecture
- 32:25Job server and clusters
- 33:49Task and workers distribution
- 35:19Logging and monitoring
- 36:48Performance and protocols
- 37:52State server and metrics
- 39:17State server benefits
- 40:01Summary and future plans
related talks
52mApache Beam, a unified model for Big Data processing
21mApache Beam (incubating)
47mApache Beam - what do I gain?
42mApache Beam: A Unified Model for...
29mApache Beam Making Big Data Portable With gRPC
39mApache Beam and Google Cloud DataFlow
10mDemocratizing the pipeline with Apache Beam and Dataflow
24mApache beam and google cloud dataflow