Unified APIs for Batch and Stream Processing on Flink
summary
Flink adds a new execution.runtime.mode toggle (streaming / batch / automatic) and modernizes time semantics so event time is the default and watermarks jump from -inf to +inf for bounded runs, avoiding late-data complexity. Batch mode switches execution to staged, one-pass algorithms (sort + group) with no checkpointing and supports existing state/timer APIs. A 14 GB benchmark at parallelism 100 drops runtime from ~1m20s (streaming) to 38s (batch). Proposal details live in FLIPs.
AI-written summary · about
chapters (10)
- 0:00Introduction to unified APIs
- 3:37Current API challenges
- 7:06Batch vs stream processing
- 11:18Goals for unification
- 15:30Modernizing data stream semantics
- 17:16Introducing batch execution mode
- 20:03Execution differences explained
- 25:14Implementation details
- 26:18Performance comparisons
- 28:27Future work and summary
related talks
43mUnified stream and batch processing with Apache Flink’s relational APIs
23mData Mesh and Data API
23mDeliver real-time data to customer using Streaming APIs
23mBuilding AI-Driven Real-Time Lakehouse APIs
27mOptimize Your API with Event-Driven Microservice Architecture.
12mOwnership: A Distributed Futures System for Fine-Grained Tasks
21mNext-Level API Management: Ensuring Data Security & Efficiency
25mAccelerate Event Enabling the API Economy