Improve your App UX with one model processing insights from batch and streams

on YouTube ↗44m

" summary

Shows a concrete pipeline that uses Apache Beam and the Cloud Dataflow runner with Pub/Sub input and BigQuery storage to run the same code for both batch and streaming. Implements a live demo game (four buttons, assigned color), processes ~41,000 events, windows at five minutes, emits early firings every 10 seconds, allows 10-minute late arrivals and uses accumulation for refinements. Deployment hits a pipeline-name collision but succeeds; batch job reuses identical transforms, proving one model handles correctness, latency and live visualization.

AI-written summary · about

" chapters (45)

1. 0:00Introduction
2. 1:04Steps to improve user experience
3. 2:10History of data processing at Google
4. 3:31What is MapReduce
5. 4:10The problem with MapReduce
6. 4:55From Java
7. 5:43The problem
8. 6:44Artificial splitting
9. 7:15Un unbounded data
10. 7:29Delays
11. 8:08How to deal with delays
12. 8:19MillVia
13. 8:55Timebased windows
14. 9:54Session windows
15. 10:06Event vs processing time
16. 11:16Stream vs Batch
17. 11:41Billing Pipeline
18. 12:53User Experience
19. 13:47Abuse Detection
20. 14:43Historical Systems
21. 15:57Apache Beam
22. 16:55Dataflow Example
23. 17:44Four Questions
24. 19:33MapReduce
25. 22:48When to omit results
26. 23:03Create a window
27. 23:25Wait for results
28. 23:58When to trigger
29. 24:30Triggers in Beam
30. 25:13Demo
31. 26:30refinements
32. 29:14how
33. 32:36what just happened
34. 33:19cancel pipeline
35. 33:54run on
36. 34:52update pipeline
37. 35:27QR code
38. 36:04Assign color
39. 36:33Running the pipeline
40. 38:10Patch pipeline
41. 38:55BigQuery
42. 39:18Color Smash
43. 41:17Hit Ratio
44. 41:31Aggregate
45. 42:49Back to the slides

" related talks

• 48mData Streaming with Apache Beam / Cloud DataflowGeoffrey Garnotel · Devfest Lille 2018
• 38mData processing with Apache BeamSourabh Bajaj · PyData Seattle 2017
• 16mYour Data Pipeline with Apache BeamSouhaib Guitouni · Devoxx France 2021
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016
• 42mApache Beam: A Unified Model for...Kenneth Knowles · Flink Forward Berlin 2016
• 53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]Markku Lepisto · PyCon APAC 2018
• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019