Apache Beam, a unified model for Big Data processing

on YouTube ↗52m

" summary

Apache Beam is presented as a unified programming model for Big Data batch and streaming processing. The talk explains Beam’s history from the Dataflow model to the present, highlights features like late data arrival handling, and shows that the same pipelines can run on on-prem engines such as Apache Spark or Apache Flink or in the cloud on Google Cloud Dataflow without changing code.

AI-written summary · about

" chapters (17)

1. 0:00Apache Beam intro
2. 1:36Big Data programming evolution
3. 4:50Batch vs streaming limits
4. 6:13Lambda architecture challenges
5. 7:22Portability and semantics
6. 10:54Dataflow unification
7. 14:12Key data processing questions
8. 16:18Beam windows, watermarks and triggers
9. 24:46Beam model deep dive
10. 31:09Beam pipelines, PCollections and Transformations
11. 37:53Beam IO connectors
12. 43:32Java pipeline example
13. 43:56Runners role and function
14. 46:23Runner compatibility
15. 48:07Use cases and applications
16. 48:49Why Apache Beam
17. 50:11Current state and outlook

" related talks

• 41mApache Beam: Unified Data ProcessingAlexey Diomin · Voxxed Days Minsk 2018
• 21mApache Beam (incubating)Eugene Kirpichov · Data By the Bay 2016
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 42mApache Beam: A Unified Model for...Kenneth Knowles · Flink Forward Berlin 2016
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018