Run Interactive Queries with Apache Flink

on YouTube ↗21m

" summary

Flink adds an intermediate-result cache (FLIP-36) that persists blocking partitions to the shuffle service so repeated sub-DAGs avoid recomputation; a new table.cache API marks results to keep across jobs in the same session. The client records an intermediate-result ID and location; if the cache is missing, the job resubmits and rebuilds the result. Work in progress includes cross-session sharing, automatic caching at shuffle boundaries, and a table.collect API.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 0:49What is interactive programming
3. 2:12Why it matters for Flink
4. 5:04Problem with recomputation
5. 7:10Solution with FLIP-36
6. 9:59How caching works
7. 15:17Future improvements
8. 17:46Q&A session

" related talks

• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache IcebergMarton Balassi, Peter Vary · Community Over Code EU 2024
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 37mThe Future of Apache FlinkAljoscha Krettek · Flink Forward Berlin 2016
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 36mInteractive Session: Running Flink SQL jobs in productionMarton Balassi, Gyula Fora, Matyas Orhidi, Erik Beebe · Flink Forward Global 2021
• 36mFlink SQL: Unified Streaming and batch for hive integrationRui Li, Jingsong Lee · Flink Forward Global Virtual 2020