Characterizing and Optimizing Remote Persistent Memory with RDMA and NVM

on YouTube ↗12m

" summary

Systematically measures RDMA accessing real production NVM (Intel 3D XPoint) and extracts nine concrete optimization hints across configuration (e.g. disable DDIO to avoid slow random NVM writes), access pattern (e.g. use 64-byte granularity to avoid PCI partial writes) and RDMA-aware techniques (e.g. combine double-batching and outstanding requests to achieve a one-round-trip persistent write). Applies optimizations to DrTM+H and Octopus, raising throughput up to 2.4X while showing some hints (like h3) can degrade performance depending on workload.

AI-written summary · about

" chapters (10)

1. 0:00Intro
2. 0:26RPM*: a key pillar in datacenter systems The key building block in modern datacenter systems
3. 1:38Challenge: how to integrate RDMA w/ NVM? Good news: RDMA can directly access NVM as DRAM Problem: the performance is poor w/o proper optimizations Leads to inefficiency in existing RDMA-NVM systems
4. 5:39Disable DDIO for one-sided RDMA DDIO: Data Direct I/O, one-sided RDMA first write to LLC However, LLC flushes to NVM in a random way & NVM has poor random access performance
5. 6:51use 64B for small writes (i.e., 256B) How to saturate NVM processing rate? • Avoid read-modify-write le.g., PCle partial write One-sided RDMA should use 64B to avoid PCle partial write
6. 7:32optimization hints (H1-H9) Focuses on remote persistent memory write
7. 7:42H9: Use OR & Doorbell for persistent write Problem: persistent one-sided RDMA needs 2 RTTU Applying RDMA-aware optimizations can reduce to 1 RTT Outstanding Request ORII
8. 8:26How to apply hints to existing systems? We use two systems as case studies Leads to 1.2-2.4x performance boost
9. 8:39Case study: DrTM+H A distributed database designed for RDMA & NVM
10. 9:56Summary & hints to future hardware designers 9 optimizations to better leverage RDMA-NVM

" related talks

• 23mTeRM: Extending RDMA-Attached Memory with SSDJiwu Shu, Zhe Yang, Youyou Lu, Qing Wang, Xiaojian Liao, Keji Huang · 22nd USENIX Conference on File and Storage Technologies (FAST '24)
• 14mUnderstanding RDMA Microarchitecture Resources for Performance IsolationJitendra Padhye, Xinhao Kong, Jingrong Chen, Wei Bai, Yechen Xu, Mahmoud Elhaddad, Shachar Raindel, Alvin R. Lebeck, Danyang Zhuo · NSDI 2023
• 18mRDMA is Turing complete, we just did not know it yet!Marco Canini, Simon Peter, Waleed Reda, Dejan Kostić · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)
• 15mCollie: Finding Performance Anomalies in RDMA SubsystemsZhuo Jiang, Jianxi Ye, Chuanxiong Guo, Xinhao Kong, Danyang Zhuo, Yibo Zhu, Huaping Zhou · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)
• 15mReplicating Persistent Memory Key-Value Stores with Efficient RDMA AbstractionJing Wang, Jiwu Shu, Youyou Lu, Qing Wang · OSDI 2023
• 16mOneSidedMW: Managing Disaggregated Memory Efficiently, Flexibly, and Securely with RNICYubin Xia, Xingda Wei, Zixuan Wang, Jinyu Gu · NSDI '26
• 19mFast Distributed Transactions for RDMA-based Disaggregated MemoryYu Zhang, Xiaofei Liao, Hai Jin, Haikun Liu, Zhuohui Duan, Haodi Lu, Yujian Zhang · USENIX ATC '25
• 06mRDMA Programming Design and Case Studies - For Better Performance DistributedYoshiro Yamabe · Open Source Summit North America 2018