Securing RDMA for High-Performance Datacenter Storage Systems

on YouTube ↗15m

" summary

RDMA adoption at Microsoft Azure and Alibaba delivers microsecond latencies and millions of bytes/sec but introduces concrete risks: NIC-side cpu bypass makes reads invisible to server CPUs so there are no server-side logs for audit, memory registrations expose large address ranges, and recent hardware shows r key non-randomness. Prior weaknesses include plaintext packets, missing integrity, and PFC denial-of-service. Paper analyzes Pilaf, FaRM and others, quantifies attack surfaces from compromised clients, and argues for NIC crypto, centralized key distribution (e.g., Zookeeper), finer-grained registrations and new abstractions for locking and isolation.

AI-written summary · about

" chapters (12)

1. 0:00Intro
2. 0:23Remote Direct Memory Access (RDMA) does CPU-bypass over the datacenter network with only a few microseconds of latency RDMA over Converged Ethernet (ROCE 2)packet
3. 2:26Example RDMA System: Pilaf (2013): Put (SEND)
4. 3:03Pilaf (2013): Unlike Put, Get is CPU-bypassing Clients Server CPU Memory
5. 3:37RDMA not designed for datacenter security needs Security weaknesses discovered over past 2 decades (see Section 2 of paper for citations)
6. 4:41We analyzed recent distributed storage systems built on RDMA and discovered additional systems design challenges even after security fundamentals are fixed.
7. 5:20Threat Model = Compromised Storage Client Server
8. 6:40no auditability/logging on reads
9. 8:36Design Implications of Storage Logic Location: RPC and Concurrency
10. 10:06Separating different users' data
11. 12:26Ingredients for more secure CPU-bypass systems
12. 13:29Lots of big open questions for future research!

" related talks

• 20mEmpowering Azure Storage with RDMAJeff Cohen, Xin Liu, Ankit Kumar, Albert Greenberg, Ivan Lee, Jitendra Padhye, Aaron Ogus, Manish Gupta, Rohan Mehta, Yuemin Lu, Wei Bai, Mahmoud Elhaddad, Shachar Raindel, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, Rebecca Chow, Vivek Ette, Igal Figlin, Daniel Firestone, Mathew George, Ilya German, Lakhmeet Ghai, Eric Green, Randy Haagens, Matthew Hendel, Ridwan Howlader, Neetha John, Julia Johnstone, Tom Jolly, Greg Kramer, David Kruse, Erica Lan, Avi Levy, Marina Lipshteyn, Chen Liu, Guohan Lu, Xiakun Lu, Vadim Makhervaks, Ulad Malashanka, David A. Maltz, Ilias Marinos, Sharda Murthi, Anup Namdhari, Madhav Pandya, Douglas Phillips, Adrian Power, Suraj Puri, Jordan Rhee, Anthony Russo, Maneesh Sah, Ali Sheriff, Chris Sparacino, Ashutosh Srivastava, Weixiang Sun, Nick Swanson, Fuhou Tian, Lukasz Tomczyk, Vamsi Vadlamuri, Alec Wolman, Ying Xie, Joyce Yom, Lihua Yuan, Yanzhao Zhang, Brian Zill · NSDI 2023
• 25mRDMA @ScaleJitu Padhye · Networking @Scale Summer 2022
• 12mPreview: NetworkingAkshay Narayan · OSDI '21 / USENIX ATC '21
• 14mUnderstanding RDMA Microarchitecture Resources for Performance IsolationJitendra Padhye, Xinhao Kong, Jingrong Chen, Wei Bai, Yechen Xu, Mahmoud Elhaddad, Shachar Raindel, Alvin R. Lebeck, Danyang Zhuo · NSDI 2023
• 27msRDMA -- Efficient NIC-based Authentication and Encryption for Remote Direct MemoryTorsten Hoefler, Konstantin Taranov, Adrian Perrig, Benjamin Rothenberger · USENIX ATC '20
• Harmonic: Hardware-assisted RDMA Performance Isolation for Public CloudsShin haong · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)
• 28mPersistent Memory plus RDMA, new age remote deviceXiao Yang · linux.conf.au 2022
• 18mRDMA is Turing complete, we just did not know it yet!Marco Canini, Simon Peter, Waleed Reda, Dejan Kostić · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)