Stream Processing End to End with Go
summary
Apache Beam now supports Go end to end and Google Cloud Dataflow accepts Go pipelines with the 2.40 release, enabling real stream processing in Go. Examples show 10-minute fixed windows with one hour allowed lateness, an early trigger that fires every 10 elements, late firings on any late data, and accumulating panes to recompute outputs. The model relies on watermarks to decide on-time vs late, runs on Dataflow or other runners (Flink/Spark), and is production-ready.
AI-written summary · about
chapters (13)
- 0:00Opening banter
- 1:18Coffee banter
- 4:47Virtual audience engagement
- 6:37Presentation introduction
- 8:23Apache Beam go SDK news
- 9:29Core pipeline concepts
- 10:24Challenges of streaming
- 12:41Windowing strategies
- 15:53Handling data lateness
- 17:12Understanding watermarks
- 19:23Triggers and pane accumulation
- 23:15Beam project overview
- 25:39Q&A and closing
related talks
39mApache Beam and Google Cloud DataFlow
16mYour Data Pipeline with Apache Beam
47mApache Beam - what do I gain?
41mGoogle Dataflow: New open model for batch & stream processing
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
26mBeam and dataflow: a perfect combination of parallel processing
46mBuilding an Apache Beam streaming runtime in GCP
24mApache beam and google cloud dataflow