Apache Fluss: A Streaming Storage for Real-Time Lakehouse
summary
Apache Fluss is a distributed streaming table store that promises millisecond reads and writes, mutable primary-key tables that emit append-only change logs, and a tiered design that writes hot data locally and persists to S3/Iceberg for months or years. Writes append zero-copy Arrow IPC column batches to log segments; default durability is 3-replica leader+2 followers. Clients perform a union read merging Iceberg snapshot + Fluss change log (snapshot then resume from log offset) to get consistent, real-time materialized views. Benchmarks claim 10x throughput when projecting 10% of columns.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:03What is Flink streaming
- 7:51Storage challenges for Flink SQL
- 9:18Introducing Apache Flux
- 14:05Flux in storage matrix
- 16:57Flux architecture overview
- 21:38Log table write path
- 23:51Primary key table writes
- 29:19Columnar streaming design
- 33:20Lakehouse with Flux
- 37:18Unified table view
- 43:48Union read for queries
- 47:47Merge on read & deletion vectors
- 53:10Metadata sync with Iceberg
- 54:17Flux roadmap
- 54:58Q&A
related talks
40mThe Seven Deadly Sins of Streaming
42mStream Processing with Apache Flink
31mFrom Warehouse to Lakehouse: Data at Rest
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg
43mUnified stream and batch processing with Apache Flink’s relational APIs
42mStreamhouse Architecture with Flink and Paimon
34mApache Iceberg.
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL