Real-time Data Processing with Cloud Dataflow

on YouTube ↗41m

" summary

Covers using Google Cloud technologies—especially Cloud Dataflow—to process large volumes of real-time data for games, apps, and IoT sensor scenarios. Presents architectures and Google Cloud tools that simplify building real-time data-processing solutions.

AI-written summary · about

" chapters (11)

1. 0:00Introduction to processing
2. 3:00What is real-time processing
3. 8:05Apache Beam model
4. 11:58Transformations and collections
5. 16:32Windows and aggregations
6. 19:44Watermarks and triggers
7. 26:22Late events and handling
8. 31:23Data accumulation and discard
9. 32:52Cloud Dataflow platform
10. 34:00Demo: Traffic simulator
11. 39:19Resources and next steps

" related talks

• 47m'No One at Google Uses MapReduce Anymore' - Cloud DataflowMartin Gorner · DevFest Nantes 2014
• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019
• 44mBig Data Science on Google CloudFernando Ultremare · GDG DataFest 2018
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 26mETL pipeline with GCP Dataflow​Oleksandr Reshetnik · Devoxx Ukraine 2021
• 47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-diveSergei Sokolenko · Highload fwdays'19
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016
• 38mShall We Talk About Data Pipelines?Cesar Nogueira · GDG DataFest 2018