Data enrichment patterns with Apache Flink

on YouTube ↗27m

" summary

Compares five concrete enrichment patterns implemented with Apache Flink and shows what actually works: preload tiny static maps into each subtask memory for 1,700 airport codes, use Flink keyed state for partitioned caches and TTL, call external models via ordered vs unordered async IO, or stream reference updates with Debezium + Kafka CDC into MapState and coProcessFunction. Demo uses Aurora MySQL → Debezium → Kafka → Flink, handles late orders by selecting the rate valid at event time, and shows tradeoffs in latency, throughput and operational complexity.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to data enrichment
2. 1:15Streaming data enrichment context
3. 2:45Caching reference data in tasks
4. 5:13Partitioned caching in Flink state
5. 8:35Synchronous and async lookups
6. 12:52TTL caching with state
7. 15:45Streaming reference data as events
8. 17:55Demo overview and setup
9. 23:14Demo code walkthrough
10. 26:03Summary and trade-offs

" related talks

• 50mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Jfokus 2025
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 39mDeclarative stream processing...Till Rohrmann, Fabian Hueske · Flink Forward Berlin 2016
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 43mContinuous SQL with Kafka and FlinkTimothy J Spann · JCON WORLD ONLINE 2023
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015