Floe: Policy-Based Table Maintenance for Apache Iceberg
summary
Open-source Floe turns Iceberg maintenance into declarative policies that run via Spark or Trino and connect to 7 catalogs (REST, Polaris, Lakekeeper, Gravitino, DataHub, Hive Metastore, Nessie). Policies use glob patterns, schedules and health triggers based on actual metrics (small-file percentage, snapshot count, delete-file ratio, partition skew), compute a debt score (10x/3x/1x weights) to rank urgency, and submit one job per operation; demo runs on 5 example tables and caches reports with a 5-minute TTL.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 7:58Iceberg table maintenance challenges
- 11:54Introducing Floe project
- 14:44Policy model and scheduling
- 20:21Health-driven maintenance approach
- 26:18Maintenance planner and scoring
- 30:12Demo overview and setup
- 33:27Live demo walkthrough
- 36:57Architecture and pluggability
- 37:38Q&A session
related talks
34mApache Iceberg.
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
41mTip of the Iceberg
50mDesigning a scalable data platform based on Apache Spark/Iceberg
52mApache Iceberg - An Architectural Look Under the Covers
52mSetting Up Iceberg Tables in Snowflake
31mOpen Standards and Lakehouse AI Development