Geoscience Big Data Computing with Python
summary
Geoscience Australia runs a petabyte-scale archive of 30+ years of Earth-observation imagery, organized into spatial partitions (now ~100 km cells) and indexed in Postgres so Python code can rapidly search and retrieve data. The system ingests calibrated Landsat/Sentinel products (a processing run produced 12 years when 10 were planned), exposes an API and Jupyter workflow, and uses GDAL, rasterio, NumPy, SciPy, NetCDF, Xarray+Dask and SSH-tunneled notebooks on NCI clusters. Retrievals return multi-band xarray time series (e.g. 75 dates for 2015) ready for analysis.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 2:00About Geoscience Australia
- 5:28System and System Engineering
- 7:47Big Data in Geoscience
- 9:12Satellite Imagery Data
- 10:27Satellite Animation
- 11:10Observation Big Data
- 13:46Geoscience Data Cube
- 17:16Landsat Archive
- 19:29Python Modules
- 20:20Datacube Architecture
- 22:05Jupiter Notebook
- 23:59Computing
- 25:02Product Story
- 25:35Satellite Imagery
- 26:15Spatial Temporal Imagery
- 27:35Data
- 28:18Image
- 30:26Conclusion
related talks
16mThe Open Data Cube and working towards a Digital Earth
31mOpenDataCube - Petabyte Scale Satellite Imagery Processing in Python
17mOpening a treasure trove: The Historical Aerial Photography project
36mGeoMonitoring with Python: Innovative Solutions for Latin America’s Challenges
18mearthaccess: Accelerating NASA Earthdata sci through open, collaborative development
41mSnakes in the Microwaves: How Python is Powering the Golden Age of SAR
14mSatellite data monitoring for analytical models
30mGeoscience at Massive Scale