RDMA at Cloud Scale: The OCI Experience

on YouTube ↗20m

" summary

Oracle OCI describes building production RDMA over Converged Ethernet (RoCE) fabrics that scale from 1,000 to 100,000 nodes, moving from two-tier to three-tier + superspine topologies and using 64–512‑port switches. Full bisection bandwidth and VLAN/IPv6 isolation support multi-tenancy; PFC is used only link-to-nic, ECN+DCQCN handles congestion, and collective load‑balancing avoids uplink hot spots. Major pain points are 400/800G link flaps and vendor interop; current clusters use uniform CX7 NICs.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to OCI networking
2. 2:23RDMA and cloud infrastructure
3. 5:08Architecture and bare metal
4. 8:26Network topology and scaling
5. 11:51Network properties and security
6. 14:27Congestion control strategies
7. 17:21Operational challenges and future

" related talks

• 25mRDMA @ScaleJitu Padhye · Networking @Scale Summer 2022
• 14mODRP: On-Demand Remote Paging with Programmable RDMARong Chen, Haibo Chen, Xingda Wei, Zixuan Wang, Jinyu Gu, Hongrui Xie · NSDI '25
• 22mMetaRoCE: Meta’s RDMA TransportArvind Srinivasan, Kingshuk Mandal · @Scale: Networking 2026
• 15mFlor: An Open High Performance RDMA Framework Over Heterogeneous RNICsChao Han, Dennis Cai, Xiaoliang Wang, Bo Li, Guihai Chen, Qiang Li, Chen Tian, Peng Zhang, Hongqiang Harry Liu, Qiao Xiang, Jianbo Dong, Fei Feng, Jiaji Zhu, Yixiao Gao, Lingbo Tang, Yaohui Wu, Shaozong Liu, Lei Yan, Zhongjie Wu, Zheng Cao, Jinbo Wu, Rui Miao, Yanfang Le, Haonan Qiu, Derui Liu, Zhiwu Wu, Weijie Li · OSDI 2023
• 20mEmpowering Azure Storage with RDMAJeff Cohen, Xin Liu, Ankit Kumar, Albert Greenberg, Ivan Lee, Jitendra Padhye, Aaron Ogus, Manish Gupta, Rohan Mehta, Yuemin Lu, Wei Bai, Mahmoud Elhaddad, Shachar Raindel, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, Rebecca Chow, Vivek Ette, Igal Figlin, Daniel Firestone, Mathew George, Ilya German, Lakhmeet Ghai, Eric Green, Randy Haagens, Matthew Hendel, Ridwan Howlader, Neetha John, Julia Johnstone, Tom Jolly, Greg Kramer, David Kruse, Erica Lan, Avi Levy, Marina Lipshteyn, Chen Liu, Guohan Lu, Xiakun Lu, Vadim Makhervaks, Ulad Malashanka, David A. Maltz, Ilias Marinos, Sharda Murthi, Anup Namdhari, Madhav Pandya, Douglas Phillips, Adrian Power, Suraj Puri, Jordan Rhee, Anthony Russo, Maneesh Sah, Ali Sheriff, Chris Sparacino, Ashutosh Srivastava, Weixiang Sun, Nick Swanson, Fuhou Tian, Lukasz Tomczyk, Vamsi Vadlamuri, Alec Wolman, Ying Xie, Joyce Yom, Lihua Yuan, Yanzhao Zhang, Brian Zill · NSDI 2023
• 12mPreview: NetworkingAkshay Narayan · OSDI '21 / USENIX ATC '21
• 16mMetaRoCE: From Spec to NIC to Open SourceBalakrishnan Raman, Sandeep Nagaraj · @Scale: Networking 2026
• 18mRoCE BALBOA: Service-Enhanced RDMA Offload Engine for Data Center SmartNICsGustavo Alonso, Maximilian Jakob Heer, Benjamin Ramhorst, Yu Zhu, Luhao Liu, Zhiyi Hu, Jonas Dann · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)