Data Stream Processing Concepts and Implementations
summary
Overview of concrete stream-processing choices and tradeoffs: Google Cloud Dataflow API style watermarks and triggers balance latency vs correctness (example: fixed 3s vs adaptive heuristics), three accumulator modes (discarding, accumulating, accumulating with retract) determine how partial window results are emitted and merged. Implementations compared: Spark uses micro‑batches (10s+) moving toward Structured Streaming, Apache Flink offers native event‑time, watermarks and low latency with RocksDB state backend and checkpointing, Kafka Streams is a lightweight library using Kafka for streams-table duality and local state, supporting at‑least‑once by default.
AI-written summary · about
chapters (7)
- 0:00Start
- 2:36Sources for Streaming Data
- 7:10Native Streaming
- 8:18Event time vs processing time
- 10:39Tumbling & Siding Windows
- 11:50Session Window
- 27:12Lookup - Local Read
related talks
44mData Streaming: Which horse to bet on?
32mBig Data Processing : Where do we go next ?
31mHelp I need a stream processor - learning how to chose ...
20mStreaming Processing Options in Google Cloud
31mBlueprints for common stream processing use cases with Flink
27mNavigating Streaming Infrastructure
01h 14mPANEL - Streaming.
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...