Apache Flink deep-dive #bbuzz
summary
Apache Flink is a streaming dataflow engine that runs batch jobs as finite streams, implements barrier checkpoints (a Chandy‑Lamport variant) to get exactly‑once semantics without pausing pipelines, and uses a JVM managed memory segments approach with pregenerated binary serializers to avoid GC blowups. It supports Hadoop formats, Kafka, Yarn and a Cloud Dataflow runner, exposes iterative feedback and delta iterations, and reports scaling matrix factorization to 28 billion entities on modest clusters using in‑place state and graceful disk spill.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 3:08What is Flink
- 8:24Use cases
- 10:35Native support
- 12:17Batch processing
- 13:06Corner points
- 16:22Program life cycle
- 19:04Data streaming analysis
- 20:24Streaming
- 24:31Algorithm
- 27:08Batch pipelines
- 28:15Batch processing vs stream processing
- 31:57Operator lifetime
- 32:59Operators
- 33:36Manage memory
- 34:28Binary data
- 35:18Net effect
- 36:06Feedback
- 36:47Matrix Factorization
- 40:35Master failover
- 41:17Flink Forward
related talks
38mApache Flink: A Very Quick Guide
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
45mApache Flink — Real-Time Data Processing
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
41mBeyond the Hype: When Apache Flink Solves Real Problems
43mA Data Streaming Architecture with Apache Flink #bbuzz
36mApache Flink in the Cloud-native Era
42mStream Processing beyond Streaming Data