Designing a scalable data platform based on Apache Spark/Iceberg
summary
Blueprint for a scalable platform that pairs Apache Spark (batch + Structured Streaming) with Apache Iceberg table format on S3, using AWS EMR/managed runtimes, Glue as the Hive metastore, Airflow for orchestration and Trino/Athena for low‑latency serving. Key claims: Iceberg restores ACID, time travel and metadata driven reads; Spark handles ingestion, transformation and streaming; automated compaction to ~1 GB files prevents millions of tiny files; a layered raw/standardized/curated layout plus OpenLineage and quality checks enforces discoverability and governance.
AI-written summary · about
chapters (8)
- 0:00Data platform evolution
- 3:30Modern architecture shifts
- 12:18Apache Spark overview
- 15:20Designing for batch analytics
- 26:28Governance and metadata
- 32:22Performance optimization
- 36:11Streaming and maintenance
- 42:05Platform orchestration and Q&A
related talks
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
25mHow to build a platform for AI and analytics based on Apache Iceberg
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
34mApache Iceberg.
21mDataEngBytes 2023 - MEL-T1-05 - Fabio Ramos
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
39mFlexible OpenSearch Data Management With Apache Iceberg: Data...