A Gentle Introduction to Apache Beam
summary
Claims that Apache Beam unifies batch and streaming with a single programming model and portable runners, demonstrates SDKs for Python Java Go SQL and Scala, and shows concrete tooling: beam playground web UI and the beam starter repo for local runs. It walks a mobile-game example using event time windows watermarks triggers and accumulation to handle late arrivals. Cited production scale: 4 trillion events daily across 3,000+ pipelines with reported 2× cost savings.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:58Lambda architecture limits
- 6:14Unified Beam model
- 9:14Beam origins and evolution
- 12:09Beam programming model
- 15:23Key Beam model questions
- 16:47Building Beam pipelines
- 22:32Beam today and portability
- 23:57Getting started with Beam
related talks
38mData processing with Apache Beam
48mData Streaming with Apache Beam / Cloud Dataflow
36mMachine Learning with Apache Beam
44mImprove your App UX with one model processing insights from batch and streams
15mApache Beam and Google DataFlow IRL with Scala Scio
01h 03mBig Data Pipelines With Scala - Part 1 of 2
19mDemystifying Modern Data Pipeline Architecture
50mA Journey in Space with Apache Kafka data streams from NASA