Connecting Apache Flink to the World...
summary
Survey of Flink connectors, with concrete internals: Kafka and Kinesis offer exactly-once for sources by integrating consumer offsets with Flink checkpoints but Kafka writes remain at-least-once today; Kafka 0.8 needs custom fetcher threads while Kafka 0.9 consumer simplifies rebalancing; file sinks use bucketing and rely on Hadoop 2.7 truncate to roll back partial files; Cassandra sinks provide exactly-once only for idempotent updates; Apache Bahir hosts many community connectors for faster releases.
AI-written summary · about
chapters (10)
- 0:00Introduction and setup
- 0:24Overview of Flink connectors
- 7:13Kafka connector deep dive
- 14:25Exactly-once semantics in Flink
- 20:36End-to-end exactly-once guarantees
- 21:59File monitoring source explained
- 23:25Rolling file sink features
- 27:43Future connector developments
- 29:51Apache Bahir connectors
- 31:40Q&A session
related talks
41mBeyond the Hype: When Apache Flink Solves Real Problems
38mApache Flink: A Very Quick Guide
27mNavigating Streaming Infrastructure
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
36mFlink SQL: Unified Streaming and batch for hive integration
42mApache Flink deep-dive #bbuzz
34mIntegrating Flink into Your Ecosystem - How to Build a Flink Connector from Scratch
50mIntroduction to Apache Kafka