ETL pipeline with GCP Dataflow​

on YouTube ↗26m

" summary

Builds an operational data store and ETL pipeline on Google Cloud using Dataflow + Apache Beam to extract from SQL/NoSQL/REST, write JSON-lines to Cloud Storage → BigQuery, and serve Tableau/Data Studio. Uses a sliced architecture: many simple source-to-cloud-storage pipelines plus separate cloud-storage-to-BigQuery loaders, a Beam-based integration framework with templates and custom transforms so new teams can onboard without code. Prototyping drops from weeks to days, the solution is serverless and successful in production.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 0:56Intro
3. 14:10Aggregation Transformation
4. 15:18Java Code
5. 23:51Outcomes

" related talks

• 30mData Warehouse in Google CloudIgor Khrol · GDG DevFest Ukraine 2017
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 38mShall We Talk About Data Pipelines?Cesar Nogueira · GDG DataFest 2018
• 01h 28mA Guide to the Modern Data StackAya Khaled, Essam Ammar · DSC MENA 2024
• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016