Marrying engineering rigor & scientific rigor for the planet
summary
Argument: Earth data needs a storage layer with database-style rigor. Earthmover PBC builds IceChunk, an open-source Apache‑2.0, Rust-core, Python-bound n-dimensional chunked array engine that adds transactions and isolated sessions, Git-like snapshot versioning, and virtual datasets that map chunks to byte ranges of existing files. It sustains cloud-scale throughput (demonstrated quarter‑million S3 reads/sec and near‑line-rate network saturation) and already powers public-sector pilots (NOAA/NWS forecasts, ARIA/NASA interest) to enable safe concurrent updates and incremental downstream compute.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 3:09Ocean modeling to open data
- 5:13Earth science data upgrade
- 10:05Meteorological data history
- 13:56Limitations of standards
- 16:44Cloud-native data vision
- 17:26IceChunk storage engine
- 20:33IceChunk performance
- 21:55Formal methods in engineering
- 23:21Public sector adoption
- 25:48Planetary data marketplace
- 30:02Future directions
- 32:07Summary
related talks
39mDatacubes: Enabling Space/Time Analytics and AI
50mDesigning a scalable data platform based on Apache Spark/Iceberg
11mBreaking Data Silos: Enterprise Data Management
25mPlatform Engineering for Modern Data Infrastructure
31mOpen Standards and Lakehouse AI Development
36mThe MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
25mHow to build a platform for AI and analytics based on Apache Iceberg
08mRevolutionizing Data Engineering for Space Exploration