Petabyte CephFS for Satellite Data Processing
summary
A CephFS cluster ingests about 7 TB/day and targets ~30 PB total, built on HP and Supermicro nodes, OpenStack VMs, Ansible for orchestration and Mellanox networking, reaching ~3.5 GB/s aggregate on 10 GbE. Flash for the metadata pool drops latency from 60 ms → 278 µs (≈×228) and striped pools plus directory-to-pool binding are used to control durability and performance. Surprises: a 100× algorithm speedup, frequent disk and cooling failures, metadata fragmentation that required manual object repair, and no practical full backup strategy. Security pain point is pool-level access decoupled from POSIX permissions.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 8:38Point clouds
- 18:43Challenges
- 25:52Gross Net Ratio
- 27:28Flash Storage
- 28:25Usage Patterns
- 30:18Cluster Size Change
- 31:35Problems
- 33:22Use cases
- 37:11Current solution
- 39:36Set File System
- 40:12Resilience
related talks
34mNVMeoF in Ceph - What's New and Performance Best Practices
46mUnified Cluster Federation: A Hyperscaled Architecture of...
42mUnlimited Fileserver with Samba CTDB and CephFS
24mHigh-Scale Networking for ML Workloads With Cilium
28mCeph RBD Integration with DPUs
14mCETOFS: A High-Performance File System with Host-Server Collaboration for Remote Storage
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
38mCeph NVMe-of Road Map