Backing up CephFS

on YouTube ↗40m

" summary

CERN backs selected CephFS shares with restic plus an orchestrator, avoiding snapshots for performance reasons; a 2 PB dataset contains roughly 400 million files, so full tree walks trigger MDS cap-release alerts and take days. Teams use per-share or per-user job splitting, recursive mtime heuristics, rsync/hardlink chains, rclone, Bacula/TSM/tape and S3-mounted FUSE to TSM. Newer Ceph releases expose a snapdiff API (and planned byte-range diffs) that may make snapshot-based, MDS-friendly incremental backups practical.

AI-written summary · about

" chapters (12)

1. 0:00Introduction and setup
2. 1:04Current backup approach at CERN
3. 3:33Audience experiences and challenges
4. 14:34Backup targets and strategies
5. 16:28Snapshot-based backup methods
6. 19:29Restore experiences and testing
7. 21:02Server-side vs user-exposed backups
8. 24:13Backup frequency and retention
9. 27:06Challenges in CephFS backups
10. 31:57Snapshot performance and improvements
11. 35:56Final thoughts and recommendations
12. 38:47Q&A and closing remarks