Ceph Messenger Situation and Optimization
summary
Integrates a UCX-based messenger into Ceph to replace the old RDMA path, implementing zero-copy active-message receive and distinct small/large send paths, per-worker connection maps that remove global locks, and multi-rail and preposted buffers for higher throughput. Tests on 100 Gbps hardware with 4 KB payloads and QD=32 show UCX reaches NIC limits at 32 concurrency vs RDMA at 64, a cluster RBD test reports ~20% throughput uplift, while a microtest vs TCP shows about 10% improvement; latency and tail metrics are not measured.
AI-written summary · about
chapters (13)
- 0:00Introduction to Ceph Messenger
- 0:48Shortcomings of current RDMA module
- 3:09UCX architecture and key design
- 5:23RDMA stack and UCX integration
- 7:12Optimizations in UCX stack
- 8:42Zero-copy message receiving
- 9:28Multi-rail features in UCX
- 10:11Performance test results
- 11:36Future optimization plans
- 12:01Q&A on RDMA performance
- 13:08Q&A on latency measurements
- 14:12Q&A on UCX adoption benefits
- 16:05Q&A on pull request plans
related talks
16mBuilding A CSFQ-Inspired Transport for Switched CXL Memory Pooling
18mHydraRPC: RPC in the CXL Era
17mDRack: A CXL-Disaggregated Rack Architecture to Boost Inter-Rack Communication
23mMulti-Path Transport for RDMA in Datacenters
37mMessenger V2: The New Ceph Wire Protocol
38mCeph NVMe-of Road Map
18mLive Group Q&A Session
19mSwCC: Software-Programmable and Per-Packet Congestion Control in RDMA Engine