MetaRoCE: Meta’s RDMA Transport
summary
MetaRoCE is a clean‑sheet RDMA transport that Meta builds for clusters scaling to over a million GPUs, claiming 90% efficiency via per‑packet multipath load‑balancing, per‑path reliability and receiver‑driven rate hints that keep congestion decisions in the NIC. It runs on commodity Ethernet (EC/ECMP only), ships with a Keysight conformance suite validated from software reference through RTL emulation to production silicon, and the tests already uncover real failures like RNR hang-ups and shared‑RTO bugs that would break multipath behavior.
AI-written summary · about
chapters (14)
- 0:00MetaRoCE and AI networking
- 2:10Traditional RDMA at scale
- 3:57Intelligence at edge endpoints
- 5:22MetaRoCE design principles
- 6:23Load balancing and reliability
- 8:31Congestion control and incast
- 10:20Avoiding QP proliferation
- 11:22Open protocol and interoperability
- 12:06Compliance in protocol spec
- 13:52Testing from software to silicon
- 15:38RNR scenarios in LLM training
- 17:25Independent RTO per path
- 19:56Early compliance importance
- 20:57MetaRoCE future directions
related talks
21mScaling RoCE Networks for AI Training
18mRoCE BALBOA: Service-Enhanced RDMA Offload Engine for Data Center SmartNICs
20mRDMA at Cloud Scale: The OCI Experience
12mPreview: Networking
19mBring RoCE (RDMA over Converged Ethernet) Capability to Virtio-net Device
25mRDMA @Scale
20mEmpowering Azure Storage with RDMA
15mFlor: An Open High Performance RDMA Framework Over Heterogeneous RNICs