Jump start large window computations with Hybrid Source

on YouTube ↗30m

" summary

Bootstrapping large-window state by replaying weeks of data is solved by reading historical files from Iceberg + Kafka then switching to live Kafka without job restarts. The implementation uses Flink Source Framework (FLIP-27) and the Hybrid Source (FLIP-150) in Flink 1.14, supports timestamp-based deferred construction, preserves record type across sources, achieves high backfill throughput then drops to steady live rates, and removes manual switch logic and operational restarts.

AI-written summary · about

" chapters (6)

1. 0:00Why bootstrapping state
2. 5:41Multiple sources, one stream
3. 10:42Flink source framework
4. 13:55Hybrid Source
5. 22:37Table partitioning
6. 26:56Summary

" related talks

• 19mStreaming Event-Time Partitioning With Apache Flink and Apache IcebergJulia Bennett · Flink Forward Europe 2019
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 43mDynamic Scaling - How Apache Flink adapts to changing workloadsTill Rohrmann · Flink Forward Berlin 2016
• 40mRobust Stream Processing with Apache FlinkJamie Grier · Flink Forward Berlin 2016
• 28mStreaming Lakehouse with Kafka, Flink and IcebergZabeer Farook · apidays Singapore 2025
• 50mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Jfokus 2025
• 24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache IcebergMarton Balassi, Peter Vary · Community Over Code EU 2024
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020