Apache Iceberg ingestion with Apache NiFi
summary
Shows NiFi flows that pull from Kafka or files, convert records to Avro, write to S3 and register files in Iceberg via Trino with Starburst Galaxy used for demos. Demonstrates microbatch pattern of landing raw files then running an ephemeral external table and INSERT INTO to do format and partition rewrites. Notes a built PutIceberg processor exists but is limited to Hadoop/Polaris use. Visible artifacts include 43 flow files from one zip and 12 metadata files for six records and a large run with ~314k rows loaded.
AI-written summary · about
chapters (8)
- 0:00Introduction to Apache Iceberg
- 1:25What is Apache Iceberg
- 6:19Demo of Apache Iceberg table creation
- 9:29Introduction to Apache NiFi
- 15:08NiFi workflow basics
- 24:11Ingesting data into Iceberg with NiFi
- 32:54Iceberg table maintenance strategies
- 37:50Q&A and closing remarks
related talks
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
34mApache Iceberg.
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
41mTip of the Iceberg
52mApache Iceberg - An Architectural Look Under the Covers
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™