Big Data Processing : Where do we go next ?
summary
Claims that Streaming 2.0 can replace separate batch and streaming stacks by merging real-time and offline streaming, adding features like kappa+ backfill, push/pull operators and mixed bounded/unbounded processing. Concrete evidence: a fat-table join example drops resource needs from 6 TB to 2 TB by removing disk buffering; periodic state flushing and operator classification enable optional scaling for many jobs. References include Flink, Spark, Presto/MPP and RocksDB state backends; Uber-style implementation demonstrates feasibility.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:16Big Data Processing Models
- 1:18Execution Models
- 4:03Fundamental Difference
- 6:16Popular Streaming Engines
- 7:30Decoupling Concepts
- 8:02In Batch Processing
- 10:39Myths about Stream Processing
- 11:50Throughput & Latency
- 12:23Scale: Impact of Output Buffering
- 13:32Scale: Impact of State
- 20:10Estimating Space Requirements
- 20:49MPP Limitations
- 28:16Streaming 2.0 = Stream + Stream Unification
related talks
45mData Stream Processing Concepts and Implementations
42mStream Processing beyond Streaming Data
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
28mBest of Stream + Batch: When Streaming Analytics Isn’t
41mStreaming: an Alternative to CRUD & Batch Processing
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
16mLife is But a (Data) Stream: Building Quality Data Pipelines
25mData: Moving with the flow