Flink SQL: Unified Streaming and batch for hive integration

on YouTube ↗36m

" summary

Flink 1.11 adds Hive metastore integration so Kafka-streamed tables persist to HDFS-backed Hive tables with exactly-once guarantees via the streaming file sink and checkpoint-based writes. Partition commits support a partition-time commit policy (demo uses 30s checkpoints and 1‑minute partition delay), and temporal joins let streaming jobs enrich events from Hive dimension tables with cache TTL. Uber jars simplify connector setup. Tests show planned 1.12 file-merge work can cut checkpoint needs from 10 minutes to ~5 seconds to reduce small-file churn.

AI-written summary · about

" chapters (10)

1. 0:00Intro
2. 0:55Typical Offline Data Warehouse
3. 2:34Warehouse tools
4. 4:26Real-time Data Warehouse
5. 5:48Lambda architecture
6. 8:35Unify Computing engine
7. 12:43Details of Streaming File Sink
8. 17:29Small Files Compaction (Flink 1.12)
9. 22:58Demo - Streaming Write/Read
10. 30:27Demo - Temporal Join Hive Table

" related talks

• 35mProduction-Ready Flink and Hive Integration - what story you can tell now?Bowen Li · Flink Forward Virtual 2020
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 40mUnify Enterprise Data Processing System: Platform-level integration of Flink and HiveBowen Li · Flink Forward Europe 2019
• 37mSQL Ask-Me-AnythingBowen Li, Timo Walther, Kurt Young, Dawid Wysakowicz · Flink Forward Europe 2019
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021