Data Streaming with Apache Beam / Cloud Dataflow
summary
A concrete case study using Apache Beam on Google Cloud Dataflow to build a taxi pipeline that reads from Pub/Sub, filters and aggregates events, performs joins, and writes position and revenue results to BigQuery. It covers Beam windowing with triggers and late-data handling, autoscaling and compiled-graph optimizations, operator state requirements and runner limits, and reports a measured throughput of 5,000 events/sec.
AI-written summary · about
chapters (13)
- 0:00Start
- 2:10DATA Platform in 5 axes
- 4:33Evolution
- 6:43Batch VS Stream
- 9:14Filter
- 9:39Explode
- 11:12Aggregation
- 21:23Unified model
- 24:51Taxi play
- 27:28What Transformation/Compute
- 28:28Where Windowing
- 31:03When Streaming / Trigger
- 33:59Pipeline Taxi
related talks
38mData processing with Apache Beam
44mImprove your App UX with one model processing insights from batch and streams
26mA Gentle Introduction to Apache Beam
15mApache Beam and Google DataFlow IRL with Scala Scio
01h 03mBig Data Pipelines With Scala - Part 1 of 2
38mLessons Learned from the Migration to Apache Airflow
35mPowering Interactive Data Analysis with Google BigQuery
36mMachine Learning with Apache Beam