Turning the database inside out again
summary
Claim: combine Apache Kafka and Apache Iceberg so the event log is the canonical store, using an Iceberg proxy for Kafka that serves Kafka offsets as on-the-fly Parquet/ORC files, a reverse proxy to rehydrate Iceberg into Kafka for backfills, and offset-based indexes plus read-through caches. Result: no brittle ETL, continuous real-time + historical queries with query-time semantics, graceful performance tradeoff via partial indexes and caching, and working designs that reduce duplication and move retention/backfill control to analytic teams.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:13Martin Kleppmann’s talk
- 4:45Why Iceberg matters
- 6:08Long-term storage in Kafka
- 12:32Write ahead log as Iceberg
- 23:37Query-time semantics
- 25:02Database indexing in Kafka
- 31:44Unindexed head concept
- 35:13Combining all concepts
- 37:21Unified storage architecture
- 41:11Summary and Q&A
related talks
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
38mMelting Icebergs: Direct access to Kafka Data via Iceberg
39mFlexible OpenSearch Data Management With Apache Iceberg: Data...
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
39mApache Iceberg Merge-On-Read: Streaming CDC
34mApache Iceberg.
49mHow Apache Iceberg Really Works
41mTip of the Iceberg