Serverless datastream processing

on YouTube ↗31m

" summary

End-to-end serverless ad‑serving pipeline for connected NYC taxis built on Google Cloud: ingest via Pub/Sub at ~10,000 events/sec, stream-transform and downsample with Dataflow, persist raw stream into BigQuery (ingest up to 100,000 rows/sec) and clean with Data Prep, train a Wide‑and‑Deep TensorFlow model on ML Engine, deploy behind a REST endpoint and call it from the pipeline; observed sub‑second query lag and a finding that ~8% of taxi passes come from airports.

AI-written summary · about

" chapters (19)

1. 0:00Introduction
2. 1:51PubSub
3. 3:48Dataflow
4. 5:51Downsampling
5. 6:32Visualization
6. 7:02Taxes
7. 7:28Configuration
8. 8:32BigQuery
9. 10:50Save data to disk
10. 12:19BigQuery demo
11. 13:40Machine learning
12. 15:29Architecture
13. 16:00Data prep
14. 16:23Data cleaning
15. 20:00ML Engine
16. 22:00Deployment
17. 23:50Demo
18. 26:00Training model
19. 29:10Data flow

" related talks

• 51mRunning serverless jobs on GCPEstefania Otero · GDG DevFest Galicia 2019
• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019
• 53mIntroduction to data streamingNicolas Fraenkel · JCON 2020
• 48mUsing Open Source Tech to Swap Out Components of Your Data PipelineRustam Mehmandarov · Devoxx Belgium 2019
• 31mBuilding Real-time Pipelines with FLaNK: A Case Study with Transit DataTimothy J Spann · XtremeJ 2023
• 55mCloud Analytics: from 0 to 100,000 events per second in 50'Didier Girard · DevFest Nantes 2016
• 16mTidy Transit: Real Life Data Modeling for Public TransportationHunter Owens · rstudio::conf 2019
• 16mTidy Transit: Real Life Data Modeling for Public TransportationHunter Owens · rstudio::conf(2022)