Shall We Talk About Data Pipelines?

on YouTube ↗38m

" summary

Shows how to build automated data pipelines using Google Dataflow and BigQuery, implemented with the Dataflow SDK and Apache Beam. Describes scheduling those pipelines using cron jobs on Google App Engine to process large volumes of data.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 3:16What is Dataflow
3. 6:16Apache Beam and SDKs
4. 9:05Processing primitives
5. 12:57Java example
6. 21:44Live demo
7. 31:26Data streaming
8. 35:10Alternatives and costs
9. 37:21Closing and Q&A

" related talks

• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 26mETL pipeline with GCP Dataflow​Oleksandr Reshetnik · Devoxx Ukraine 2021
• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 10mDemocratizing the pipeline with Apache Beam and DataflowTyler Akidau, Reuven Lax · Strata Data Conference 2017 - New York, New York
• 16mYour Data Pipeline with Apache BeamSouhaib Guitouni · Devoxx France 2021
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017