Fragmentation in Ceph: Why It Happens and How We Fix It
summary
Fragmentation is a primary root cause of decade‑long Ceph performance regressions, arising in FileStore on XFS and Btrfs from inode splits, allocation group scattering, and copy‑on‑write holes and persisting in BlueStore via small overwrites and free space fragmentation. RocksDB metadata compaction behavior and RocksDB tombstones drive latency spikes until compaction triggers are added. Practical fixes include preallocating SST space, bluestore_min_alloc_size tuning, on‑clone defragmentation versus elastic shared blobs (much lower CPU and clearer clone performance), and defragment‑on‑deep‑scrub with OSD fragmentation score, IOwatcher and IOAT for analysis.
AI-written summary · about
chapters (12)
- 0:00Introduction and talk overview
- 1:03File store fragmentation causes
- 7:03BlueStore and RocksDB design
- 10:35Copy-on-write fragmentation in BlueStore
- 13:02Tombstone handling in RocksDB
- 15:54Pre-allocation for RocksDB files
- 16:58Defragmenting objects in BlueStore
- 19:47RBD mirroring performance fix
- 21:52Deep scrub defragmentation
- 23:17Compression and defragmentation
- 24:00Memory allocation in BlueStore
- 25:07Q&A on fragmentation tools
related talks
15mYears of Wear in Minutes: Benchmarking Aged Filesystems
04mCeph and NVMe/TCP
27mUnlocking Storage Efficiency: Diagnosing & Optimizing Performance in Scaling Infra
18mMedFS: Pursuing Low Update Overhead via Metadata-Enabled Delta Compression for...
40mBenchmarking: Repeatable & Comparable
21mFastCommit: resource-efficient, performant and cost-effective file system...
11mEROFS as a replacement for EXT4 and Squashfs
25mOptimizing Container Registry Storage: Introducing Blob De-Duplication for Accurate Sizing and Cost