Cloud Dataflow and the FlumeJava Paper Explained
summary
Argues that MapReduce is outdated and presents Flume and MillWheel as the foundations for a unified batch-and-stream model implemented as Cloud Dataflow. Describes FlumeJava primitives (parallel Do, groupBy, combine, flatten), explains how associative combines reduce network traffic and how an optimizer rewrites pipelines into map‑shuffle‑combine stages. Demonstrates launching a live job on 50 machines over Wikipedia logs to compute monthly top sessions and reports a result for the user "cinebot" with 103,511 edits.
AI-written summary · about
chapters (7)
- 0:00Cloud Dataflow intro
- 11:34FlumeJava primitives
- 24:07Real-world use case
- 33:24Execution graph optimization
- 41:16Dataflow deployment
- 44:11Production demo
- 51:11Q&A
related talks
42mStream Processing with Apache Flink
45mDeep dive into Unbounded Data Processing Systems
01h 06mOrchestrating Data Microservices with Spring Cloud Data Flow - Mark Pollack
50mWalking Through Spring Cloud Data Flow
19mDemystifying Modern Data Pipeline Architecture
44mData Processing With Microservices
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
01h 01mApache Fluss: A Streaming Storage for Real-Time Lakehouse