Streaming Processing Options in Google Cloud

on YouTube ↗20m

" summary

Google Cloud offers a stack for streaming: Cloud Pub/Sub for global ingestion, Apache Beam/Dataflow as a serverless runner with exactly‑once semantics when coupled with Pub/Sub, Dataproc for managed Flink/Spark, and BigQuery streaming API plus BI Engine or Sheets for real‑time dashboards. Demo shows taking a Pub/Sub topic of sales events, joining against a 50‑row state→7‑region BigQuery table, computing 5‑second fixed windows, writing aggregates into BigQuery and surfacing them in Sheets for near‑real‑time dashboards.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 4:03Dataflow
3. 4:48Apache Beam Sdk
4. 14:54Bigquery

" related talks

• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016
• 26mETL pipeline with GCP Dataflow​Oleksandr Reshetnik · Devoxx Ukraine 2021
• 55mCloud Analytics: from 0 to 100,000 events per second in 50'Didier Girard · DevFest Nantes 2016
• 41mReal-time Data Processing with Cloud DataflowFernando Ultremare · GDG Campinas Data Fest 2017
• 45mData Stream Processing Concepts and ImplementationsMatthias Niehoff · Devoxx Belgium 2017
• 28mStream Processing End to End with GoRobert Burke · GopherCon Europe 2022
• 16mYour Data Pipeline with Apache BeamSouhaib Guitouni · Devoxx France 2021