PeRF: Preemption-enabled RDMA Framework

on YouTube ↗20m

" summary

PeRF implements RNIC preemption in software using managed QPs and zero-weight work requests to break large RDMA transfers into interleaved sub-requests and to pause/resume QPs via a preemption control QP. Prototype runs on a ConnectX‑6 100Gbps NIC with a 16‑core i7 and 32GB RAM, offloads control to a single dedicated core. Paper shows RDMA without isolation causes MF throughput to fall 3.8× and DF latency to rise 6.1×; PeRF matches hardware priority schemes for latency and outperforms token-based software (Justicia) on throughput and utilization.

AI-written summary · about

" chapters (14)

1. 0:00PeRF and RDMA intro
2. 1:04Performance isolation issues
3. 3:34Apps by message/QP levels
4. 5:27RDMA hardware limits
5. 6:30PeRF solution overview
6. 7:13Preemption with managed QPs
7. 9:45Message-level preemption
8. 10:49QP-level pause/resume
9. 12:15PeRF architecture
10. 14:05PeRF evaluation setup
11. 15:09Performance vs SP and ETSS
12. 16:16Message isolation vs Justicia
13. 18:04QP isolation in apps
14. 19:32Conclusion

" related talks

• 14mODRP: On-Demand Remote Paging with Programmable RDMARong Chen, Haibo Chen, Xingda Wei, Zixuan Wang, Jinyu Gu, Hongrui Xie · NSDI '25
• 28mPersistent Memory plus RDMA, new age remote deviceXiao Yang · linux.conf.au 2022
• 12mPreview: NetworkingAkshay Narayan · OSDI '21 / USENIX ATC '21
• 14mUnderstanding RDMA Microarchitecture Resources for Performance IsolationJitendra Padhye, Xinhao Kong, Jingrong Chen, Wei Bai, Yechen Xu, Mahmoud Elhaddad, Shachar Raindel, Alvin R. Lebeck, Danyang Zhuo · NSDI 2023
• 18mRDMA is Turing complete, we just did not know it yet!Marco Canini, Simon Peter, Waleed Reda, Dejan Kostić · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)
• 15mFlor: An Open High Performance RDMA Framework Over Heterogeneous RNICsChao Han, Dennis Cai, Xiaoliang Wang, Bo Li, Guihai Chen, Qiang Li, Chen Tian, Peng Zhang, Hongqiang Harry Liu, Qiao Xiang, Jianbo Dong, Fei Feng, Jiaji Zhu, Yixiao Gao, Lingbo Tang, Yaohui Wu, Shaozong Liu, Lei Yan, Zhongjie Wu, Zheng Cao, Jinbo Wu, Rui Miao, Yanfang Le, Haonan Qiu, Derui Liu, Zhiwu Wu, Weijie Li · OSDI 2023
• 17mWhite-Boxing RDMA with Packet-Granular Software ControlMing Liu, Arvind Krishnamurthy, Chenxingyu Zhao, Jaehong Min · NSDI '25
• 18mBURST: Seeking High-performance, Interoperability and Scalability in Soft-RDMAYong Zhang, Huijun Shen, Zelong Yue, Jian Yang, Zhuo Jiang, Lang An, Yulin Chen, Luochangqi Ding, Xiaolong Zhong, Zhihong Wang, Jie Ding, Hongyu Wu, Jianxi Ye, Xijin Yin, Xingyu Zhang, Xingyu Guo, Guo Chen · NSDI '26