Data processing with Apache Beam

on YouTube ↗38m

" summary

Shows the Python SDK for Apache Beam implementing both batch and streaming pipelines that actually run on Spark Flink and Google Cloud Dataflow. Demonstrates concrete code: wordcount and a sliding five‑minute hashtag pipeline that writes to BigQuery, uses PCollections ParDo and windowing and triggers, and handles late events via a watermark and repeated firings. Describes a Dockerized execution harness and the Beam FnApi for runner portability and cites real adopters (Spotify) and a mobile‑gaming example that exists.

AI-written summary · about

" chapters (12)

1. 0:00Apache Beam intro
2. 1:26Unified data processing
3. 2:49Streaming vs batch trade-offs
4. 7:52Beam terminology
5. 10:21Event vs processing time
6. 14:40Watermarks in streaming
7. 16:33Triggers and late data
8. 18:40Word count pipeline
9. 22:19Hashtag trends pipeline
10. 25:11Beam portability
11. 30:30Beam future vision
12. 31:53Q&A

" related talks

• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018
• 48mData Streaming with Apache Beam / Cloud DataflowGeoffrey Garnotel · Devfest Lille 2018
• 26mA Gentle Introduction to Apache BeamDavid Regalado · Big Data Conference Europe 2023
• 41mTalk Python To Me: Stream Processing in your favorite language with Beam on FlinkAljoscha Krettek · Flink Forward Berlin 2017
• 42mApache Beam on Dask- Portable, Scalable, Scientific PythonAlex Merose · PyData NYC 2022
• 44mImprove your App UX with one model processing insights from batch and streamsRobert Kubis · Voxxed Days Ticino 2017
• 36mMachine Learning with Apache BeamDanny McCormick · Open Source 101 2023
• 38mBeam on Flink: How does it actually work?Maximilian Michels · Flink Forward Europe 2019