The future of Google Cloud data processing

on YouTube ↗10m

" summary

Google Cloud DataProc now runs managed open‑source components both on Yarn and on Kubernetes, shipping a Spark Operator and a Flink Operator and a unified control plane that exposes clusters, jobs and workflows APIs. Demoed job output streams driver logs directly and computes pi as 3.14, showing Spark driver and container images match across environments so SSH and manual tuning aren’t needed. Goal is portable PI Spark workloads across clouds (GCP and AWS), better component isolation, and Kubernetes primitives like autoscaling and cluster healing.

AI-written summary · about

" chapters (5)

1. 0:00Intro
2. 0:19Dataproc
3. 3:00Key points
4. 4:28Demos
5. 6:49Kubernetes Engine

" related talks

• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019
• 41mReal-time Data Processing with Cloud DataflowFernando Ultremare · GDG Campinas Data Fest 2017
• 44mBig Data Science on Google CloudFernando Ultremare · GDG DataFest 2018
• 12mState of Cloud Foundry on Google Cloud PlatformMeaghan Kjelland · Cloud Foundry Summit Silicon Valley 2017
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 04mLightning Talk: Google Cloud Data Services and Cloud FoundryEric Johnson · Cloud Foundry Summit Europe 2016
• 47m'No One at Google Uses MapReduce Anymore' - Cloud DataflowMartin Gorner · DevFest Nantes 2014