Optimizing Ceph IO for High Throughput Particle Physics Workflows
summary
65 PB Ceph cluster backing LHC analysis sustains ~20 GB/s but small vector reads against EC pools cause severe lock contention and 5+ second latencies for 100×4 KB reads. Replacing librados_striper locking with batched atomic librados reads inside xrootd-ceph (no Ceph server changes) reduces per-read latency to tenths of milliseconds, cuts timeouts and variance, and raises throughput — at the cost of assuming immutability (risk of corruption if files are modified).
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 7:23High Luminosity LHC challenges
- 14:07Small read performance issues
- 18:21Liberato striper inefficiency
- 24:19Optimizing xrd SEF for small reads
- 32:07Performance improvements and results
- 34:15Lessons and future optimizations
- 36:42Q&A
related talks
36mRapid Processing of NASA Satellite Data Stored with Ceph
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
01h 06mBest practices when accessing Big Data or any other data!
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
35mScaling Parallel Algorithms to Massive Datasets using Multi-SSD Machines
28mOptimizing Latency-Sensitive Queries for Presto at Facebook
14mFalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
29mOn Data Locality in Kubernetes