Google Dataflow: New open model for batch & stream processing
summary
Explains that Google Dataflow becomes an open model now incubating as Apache Beam and shows a working pipeline reading Wikipedia edits from Pub/Sub, counting changes with sliding windows, watermarks and triggers to emit early and late results. Demonstrates live dashboard to Firebase, stores 100 million events (≈50 GB) in BigQuery, sustains ~250 elements/sec across windows with ~9s system lag, and claims exactly‑once semantics when using Beam runners like Flink or Cloud Dataflow.
AI-written summary · about
chapters (10)
- 0:00Introduction and speaker intro
- 1:26What is Apache Beam
- 6:47History of data processing models
- 15:38Dataflow solution overview
- 25:08Dataflow programming model
- 28:45Live demo setup
- 31:37BigQuery analysis demo
- 33:26Firebase real-time results
- 35:11Apache Beam open model
- 37:37Q&A on Lambda architecture
related talks
39mApache Beam and Google Cloud DataFlow
26mBeam and dataflow: a perfect combination of parallel processing
24mApache beam and google cloud dataflow
46mBuilding an Apache Beam streaming runtime in GCP
47mApache Beam - what do I gain?
16mYour Data Pipeline with Apache Beam
21mApache Beam (incubating)
52mApache Beam, a unified model for Big Data processing