Rapid Processing of NASA Satellite Data Stored with Ceph
summary
A six-petabyte Ceph cluster using librados, EC pools (6+2 erasure coding) and CephFS stores millions of VIIRS granules and ingests ~600 GB/day while producing ~500 GB/day (peaks to ~10 TB/day possible) for multi-year reprocessing. Custom tools include PDSserve (byte-range reads of 5 GB level‑0 files) and radoslite/PGTrack for metadata and tracking. HTCondor + Kubernetes run 2,400-core jobs with sustained network I/O near 10 GB/s during processing and cluster recovery peaking ~28 GB/s; several outages occur but no object loss.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 2:39Primary instruments
- 3:33Data format
- 5:24Polar orbits
- 6:03Other satellites
- 6:57Tropic
- 7:20Tropics
- 8:33Storage requirements
- 9:18Data ingest
- 10:01RabbitMQ
- 10:41Checksum
- 11:19PGTrack
- 12:00rado sleight
- 13:03pool alignment
- 13:59PDSserve
- 15:44LiberaDOS
- 16:29Kubernetes
- 17:19POSIX
- 18:24HT Condor
- 19:53Cluster Structure
- 20:40Graphs
- 22:08Processing Workflow
- 23:08Data Axis
- 24:48FDB
- 26:22Growing pains
- 30:02Architecture
- 30:58Performance
- 31:52Scale up
- 32:50Server fail
- 33:28Future plans
related talks
36mOptimizing Ceph Object Storage for Analytics and AL/ML Workloads - Overview and Future Plans
43mOptimizing Ceph IO for High Throughput Particle Physics Workflows
41mMaximising Data Storage Efficiency with Ceph as a Caching Layer for Tape Archiving
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
05mAlex Kudriashova Interview
11mProcessing TBs of Astronomy Data at Scale with the AC3 Horizon Research Project
35mBuilding and Scaling Data Pipelines for Data Engineering With Ceph Object Storage
37mAn Initial Look at Deep Learning IO Performance