Joining Infinity: Windowless Stream Processing with Flink
summary
ResearchGate converts change data capture Kafka topics into a continuous join using Flink (coFlatMap, keyed operator state, checkpoints and savepoints) to map publications to accounts and achieve millisecond latencies in production on Flink 0.10. Deletes and updates force storing a previous-value state per input key so delete records can be routed to the correct author key; many-to-many joins break without natural IDs. The implementation requires two in-memory copies and network to reshuffle twice, but it runs reliably with fast recovery.
AI-written summary · about
chapters (13)
- 0:00Introduction to ResearchGate
- 1:08Research timeline feature
- 3:42Current technical state
- 5:05Data model for joins
- 8:19Change data capture explained
- 11:58Flink streaming join setup
- 13:27Join implementation in Flink
- 16:41Handling deletes and updates
- 21:17Natural IDs for correctness
- 24:09Generic stream join approach
- 26:56Memory and network trade-offs
- 28:21Production deployment results
- 29:33Q&A session
related talks
19mIntegrating Flink and Kafka in the standard genomics pipeline
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
27mNavigating Streaming Infrastructure
39mChangelog Stream Processing with Apache Flink
28mMigration of a realtime stats product from Storm to Flink
43mContinuous SQL with Kafka and Flink
36mFlink SQL: Unified Streaming and batch for hive integration
43mA Data Streaming Architecture with Apache Flink #bbuzz