RDMA @Scale

on YouTube ↗25m

" summary

Azure moves most storage traffic off CPUs: about 65% of data-plane traffic uses RDMA, across links up to 100 km and what the talk claims as an exabyte-scale footprint. The stack uses RoCE v2 over lossless Ethernet plus per‑flow congestion control DCQCN (CNP feedback) to avoid PFC victimization; a validated fluid model tuned router ECN and PFC thresholds, was implemented with Mellanox silicon, and the deployment at 100k‑node scale reportedly works with PFCs rare.

AI-written summary · about

" chapters (11)

1. 0:00RDMA at Microsoft
2. 1:51What RDMA is
3. 5:45RDMA in Azure storage
4. 8:35InfiniBand to Ethernet
5. 11:06Priority Flow Control
6. 13:33Victim flow problem
7. 16:05Data Center Quantized Congestion Control
8. 18:11Tuning DCQCN
9. 21:02Mathematical modeling of DCQCN
10. 22:25Deployment challenges
11. 23:53RDMA future

" related talks

• 20mEmpowering Azure Storage with RDMAJeff Cohen, Xin Liu, Ankit Kumar, Albert Greenberg, Ivan Lee, Jitendra Padhye, Aaron Ogus, Manish Gupta, Rohan Mehta, Yuemin Lu, Wei Bai, Mahmoud Elhaddad, Shachar Raindel, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, Rebecca Chow, Vivek Ette, Igal Figlin, Daniel Firestone, Mathew George, Ilya German, Lakhmeet Ghai, Eric Green, Randy Haagens, Matthew Hendel, Ridwan Howlader, Neetha John, Julia Johnstone, Tom Jolly, Greg Kramer, David Kruse, Erica Lan, Avi Levy, Marina Lipshteyn, Chen Liu, Guohan Lu, Xiakun Lu, Vadim Makhervaks, Ulad Malashanka, David A. Maltz, Ilias Marinos, Sharda Murthi, Anup Namdhari, Madhav Pandya, Douglas Phillips, Adrian Power, Suraj Puri, Jordan Rhee, Anthony Russo, Maneesh Sah, Ali Sheriff, Chris Sparacino, Ashutosh Srivastava, Weixiang Sun, Nick Swanson, Fuhou Tian, Lukasz Tomczyk, Vamsi Vadlamuri, Alec Wolman, Ying Xie, Joyce Yom, Lihua Yuan, Yanzhao Zhang, Brian Zill · NSDI 2023
• 20mRDMA at Cloud Scale: The OCI ExperienceJag Brar, David Becker · @Scale: Networking 2025
• 15mSecuring RDMA for High-Performance Datacenter Storage SystemsIrene Zhang, Jacob Nelson, Adriana Szekeres, Anna Kornfeld Simpson · HotCloud '20
• 12mPreview: NetworkingAkshay Narayan · OSDI '21 / USENIX ATC '21
• 23mMulti-Path Transport for RDMA in DatacentersGuo Chen, Peng Cheng, Yongqiang Xiong, Yuanwei Lu, Kun Tan, Thomas Moscibroda, Bojie Li, Jiansong Zhang, Enhong Chen · NSDI '18
• 18mRoCE BALBOA: Service-Enhanced RDMA Offload Engine for Data Center SmartNICsGustavo Alonso, Maximilian Jakob Heer, Benjamin Ramhorst, Yu Zhu, Luhao Liu, Zhiyi Hu, Jonas Dann · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)
• 15mFlor: An Open High Performance RDMA Framework Over Heterogeneous RNICsChao Han, Dennis Cai, Xiaoliang Wang, Bo Li, Guihai Chen, Qiang Li, Chen Tian, Peng Zhang, Hongqiang Harry Liu, Qiao Xiang, Jianbo Dong, Fei Feng, Jiaji Zhu, Yixiao Gao, Lingbo Tang, Yaohui Wu, Shaozong Liu, Lei Yan, Zhongjie Wu, Zheng Cao, Jinbo Wu, Rui Miao, Yanfang Le, Haonan Qiu, Derui Liu, Zhiwu Wu, Weijie Li · OSDI 2023
• 17mWhite-Boxing RDMA with Packet-Granular Software ControlMing Liu, Arvind Krishnamurthy, Chenxingyu Zhao, Jaehong Min · NSDI '25