Open Standards and Lakehouse AI Development
summary
Argues that combining Apache Iceberg table metadata, the Apache Iceberg REST Catalog Spec and columnar transport via Arrow Flight yields a lakehouse that is fast, interoperable and agent-ready. Shows concrete stack: Parquet files + Iceberg manifests + Polaris or REST catalogs + Arrow in-memory format + Arrow Flight transport + an MCP server for LLM/tool orchestration. Demonstrates big speedups over JDBC/ODBC at million+ to billion-row scales, and notes MCP servers (e.g., Fiverr use case) plug directly into LLMs and work.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 1:52Lakehouses and open standards
- 3:59What is a data lakehouse?
- 9:57Lakehouse architecture overview
- 10:17Apache Iceberg explained
- 14:10Iceberg REST catalog protocol
- 17:21Catalog options for Iceberg
- 18:44Apache Arrow and Arrow Flight
- 23:16Model Context Protocol for AI
- 25:43Semantic layers and AI agents
- 26:47Apache Polaris catalog features
- 27:49Live demo and resources
- 28:11Q&A session
related talks
49mHow Apache Iceberg Really Works
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
05mThe Future of Lakehouse Format Interoperability
34mApache Iceberg.
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
25mHow to build a platform for AI and analytics based on Apache Iceberg
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
50mDesigning a scalable data platform based on Apache Spark/Iceberg