Apache Beam (incubating)

on YouTube ↗21m

" summary

Apache Beam traces Google’s evolution from MapReduce/FlumeJava/MillWheel into a single programming model that unifies batch and streaming, implemented as Java and Python SDKs and runnable on Cloud Dataflow, Flink, Spark or locally. It exposes what/where/when/how for temporal processing: element transforms, event-time windowing, triggers and watermarks, and strategies for late-data refinement, providing exactly-once semantics for correctness. The project is Apache-licensed, donated with 44 unanimous votes, and is production-ready with multiple runners and an active ecosystem.

AI-written summary · about

" chapters (8)

1. 0:00Introduction to Apache Beam
2. 0:34Google's data processing evolution
3. 4:49Limitations of batch processing
4. 7:25MillWheel and streaming correctness
5. 9:51Unified batch and streaming model
6. 10:33Programming model pillars
7. 14:46Apache Beam project launch
8. 17:42Q&A session

" related talks

• 52mApache Beam, a unified model for Big Data processingJean-Baptiste Onofré, Ismaël Mejía · DevFest Nantes 2016
• 41mApache Beam: Unified Data ProcessingAlexey Diomin · Voxxed Days Minsk 2018
• 42mApache Beam: A Unified Model for...Kenneth Knowles · Flink Forward Berlin 2016
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 29mApache Beam Making Big Data Portable With gRPCIsmaël Mejía · gRPC Conf 2020
• 10mDemocratizing the pipeline with Apache Beam and DataflowTyler Akidau, Reuven Lax · Strata Data Conference 2017 - New York, New York