Ceph in Scientific Computing and Large Clusters BoF
summary
BoF reports a production Ceph front end that serves a 30 PB cluster with ~3,000 OSDs and exposes only S3, using an in‑house frontend and backend to mint 4‑hour expiring S3 keys with ACL templates for public/private buckets and rclone templates and an in‑house daemon to revoke keys automatically; teams describe upgrade paths from Nautilus→Octopus, node rebuilds via kickstart taking ~90 minutes, strategies for virtual racks to change CRUSH domains, and mixed OS rollouts that completed with minimal user impact.
AI-written summary · about
chapters (11)
- 0:00Introduction to BoF session
- 1:43Purpose of Ceph in research
- 4:13LUMI supercomputer S3 access solution
- 12:47Discussion on S3 credential management
- 15:12Cluster horror stories and data spikes
- 17:30Rolling reboots and upgrades in large clusters
- 21:06Maintenance windows and OS upgrades
- 23:36Cluster upgrade strategies and automation
- 27:55Host-level redundancy and failure domains
- 31:40Changing crush rules in production
- 37:06Summary and next steps
related talks
38mDelivering Enterprise-Grade Object Storage: Bloomberg’s IAM Journey in a Multi-Tenant Arch...
23mSMB Keeping - Clients, Clusters & Clouds
35mData Security and Storage Hardening in Rook and Ceph
30mData Security: Theoretical and Real World Approache...
18mData Security and Storage Hardening in Rook and Ceph
29mRotating Heads and CephX Keys
24mSecurity Process Improvements in Ceph and Rook
31mThe SMB Report Card