Facebook's Data Center Fabric

on YouTube ↗38m

" summary

Facebook replaces oversized cluster networks with a building-wide layer‑3 fabric made of pods (a COD = 48 racks per pod aggregated on four top-of-rack uplinks) and parallel spine planes with all inter-switch links at 40 G interswitch links today, upgradeable to 100/200/400G. BGP + ECMP handles hundreds of thousands of prefixes while a centralized controller monitors and injects paths when needed. Automation generates configs, verifies cabling (under less than 1% cabling errors in initial runs) and hammers provisioning; operators put large deployments into service fast (120 boxes deployed by two people), and the fabric is running in production.

AI-written summary · about

" chapters (35)

1. 0:00Intro
2. 0:194-post Cluster: a unit of deployment Facebook Backbone
3. 0:484-post Cluster: a [large] unit of deployment
4. 1:10Traffic growth: exponential
5. 1:33The Fabric: one datacenter-wide network
6. 3:02Server Pod: a new unit of deployment
7. 3:30Server Pod: a (small) unit of deployment
8. 4:16Distributed load, resilience to failures
9. 4:38Modularity and scalability
10. 5:06More bandwidth capacity
11. 6:16Physical design: end-state
12. 6:48Simpler sizing & capacity planning
13. 7:34Simpler components
14. 8:40Optimize power & cooling Datacenter Suite X
15. 9:59Flip side
16. 12:30Physical: data center design
17. 12:42Fabric-optimized data center
18. 13:15Logical: transition from clusters
19. 14:56Physical Clusters
20. 15:15Virtual Clusters
21. 15:49Technology: routing design
22. 16:24Technology: keep it simple stupid
23. 16:52Distributed control, centralized override
24. 17:42Config generation
25. 19:12Auto-discovery and auto-provisioning
26. 20:12Simplistic box specs
27. 20:46Basic box actions
28. 21:08The Hammer
29. 21:50Aggiornamento!
30. 22:24Configuration monitoring
31. 22:48Monitoring: heatmap
32. 23:07Monitoring: dashboards
33. 23:36FBAR: auto-remediation
34. 23:51Manage like servers
35. 24:23Fabric tooling & integration

" related talks

• 11mRunning BGP in Data Centers at ScalePetr Lapukhov, Jingyi Yang, Aditya Akella, Anubhavnidhi Abhashkumar, Kausik Subramanian, Alexey Andreyev, Hyojeong Kim, Nanda Kishore Salem, Hongyi Zeng · NSDI ’21
• 39mLoad Balancing Building BlocksKyle Lexmond · SREcon19 Europe/Middle East/Africa
• 46mLessons learned from monitoring a large production networkGiacomo Bagnoli · DevOpsDays Madrid 2020
• 37mDisaggregating the Network: Switching as a ServiceNina Schiff · LISA17
• 44mHow we designed an OpenStack Public Cloud at Scale with Fabric NetworkingAbhisak Chulya Charnsilp, Chinprasert Jeeraput · OpenInfra Summit Berlin 2022
• 25mStardust: Divide and Conquer in the Data Center NetworkNoa Zilberman, Gabi Bracha, Golan Schzukin · NSDI 2019
• 17mMeta’s Network Journey to Enable AIHany Morsy, Susana Contrera · Networking @Scale 2023
• 14mRDC: Energy-Efficient Data Center Network Congestion Relief with TopologicalWeitao Wang, T. S. Eugene Ng, Ang Chen, Dingming Wu, Sushovan Das, Afsaneh Rahbar · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)