OpenDataCube - Petabyte Scale Satellite Imagery Processing in Python
summary
A working claim: a single Python API call — datacube.load() single API — lets users query and lazily process continental petabytes of satellite imagery indexed by time/space and returned as xarray/dask arrays. Deployments use a cloud-optimized GeoTIFF (COG) backend, PostgreSQL or STAC-style catalogs, and Kubernetes + Dask to spin thousands of CPUs (examples: a 4,000‑CPU run consuming ~50 TB RAM). Known limits include SAR complex‑pixel and hyperspectral scaling and an S3 concurrency ceiling near 5.5k concurrent S3 reads.
AI-written summary · about
chapters (12)
- 0:00Start
- 2:51What is in a name?
- 3:58Origins in the bowels of HPC
- 7:41The joys of NetCDF
- 15:39YAML Engineering -The Data Model
- 17:29PostgreSQL as document database
- 18:16PostgreSQL and ORM
- 21:32Magical Load command
- 22:31Data Model Shortcomings : SAR
- 22:58Data Model Shortcomings : HyperSpectral
- 24:11Parallelism in Python - Dask way
- 28:29Enhancement Proposal - Hyperspectral Support
related talks
34mGeoscience Big Data Computing with Python
39mDatacubes: Enabling Space/Time Analytics and AI
26mWho Said Wrangling Geospatial Data at Scale was Easy?
30mDask ❤️ Xarray: Geoscience at Massive Scale
28mProcessing Large Geospatial Datasets with Dask & Xarray
30mGeoscience at Massive Scale
25mA Journey Through a Geospatial Data Pipeline: From Raw Coordinates to Actionable Insights
28mBreaking up arrays up into chunks for fun and science with Xarray and Dask