Kubernetes Meets Climate Data: Designing Analysis-Ready Dat...
summary
Built a Kubeflow-based pipeline that uses Kubeflow pipelines, DuckDB and Xarray/Zarr to turn heterogeneous EO products into analysis-ready cubes with object-store native access. It tackles a real scale problem — one use case has 1 million files / 20 TB — by batching with a FIFO forecast-wave and tiered backfill/warm/daily runs to avoid rates and consistency issues. The system enforces single-writer grouping to prevent Zarr corruption, exposes notebooks and an interactive UI that joins forecasts and Sentinel‑3 observations and loads two years in ~45s.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:54Climate data access challenges
- 8:08Kubernetes for climate data
- 12:48Concurrent writes and consistency
- 15:46Batching large datasets
- 18:13Demo: analysis-ready visualization
- 19:37Future steps and open-source
related talks
31mKubeflow - Data Science on Steroids
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
35mOrganizing data engineering around Kubeflow
27mKubeflow for enabling AI Powered Drug Discovery and development...
37mPanel: Enterprise-grade, On-prem Kubeflow in the Financial Sector
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...
51mMachine Learning using Kubeflow and Kubernetes