Cloud-Native Data in Practice
summary
The Australian Ocean Data Network moves large marine data onto Amazon S3 and experiments with cloud-native formats to avoid massive downloads; using Zarr with a 100×100×100 chunks scheme and access from a Jupyter notebook yields a 31‑hour → 1‑hour → 1‑minute progression (legacy download / kachunk index / direct Zarr) and one reported 1900× faster workload in another case. GeoParquet + columnar storage and stack catalogs enable mixing tabular/geospatial sources; persistent problems are non‑ACID multi-file updates, format immaturity and uncertain long‑term stability.
AI-written summary · about
chapters (7)
- 0:00Introduction to IMOS
- 3:08Moving to cloud-native data
- 4:11Use case grid time series
- 8:46Use case data mixing
- 11:13Format comparisons
- 13:39Challenges with formats
- 16:28Q&A session
related talks
23mLessons learned building a cloud native geospatial data management system
20mCloud Native Patterns for Geospatial Analytics and AI
22mTake your Business to a New Geography
47mCloud Native and Sustainable, Reproducible Scientific Computing
36mAn Exploration in Storing Telemetry in Cloud Object Storage
22mKubernetes Meets Climate Data: Designing Analysis-Ready Dat...
41mOn Data: Sky above, sand below, peace within
22mRunning Geospatial Workloads on AWS