Detect, Decide, Fence: Resolving Netsplits in 3-AZ Stretch Cluster
summary
Concrete netsplit resolution for 3‑AZ Ceph: monitors publish connection scores to build a location graph, the system runs Bron–Kerbosch to find maximal cliques (exponential but trivial for 3–5 AZs), then applies a cascading heuristic—total effective OSD weight, up monitor count, monitor connection sum, lexicographic tiebreaker—to pick the surviving sites. Pre‑fence safety simulates PG acting sets, fencing excludes a bucket of OSDs and sets no_recover/no_backfill, and fences lift only after a 90s lift threshold.
AI-written summary · about
chapters (8)
- 0:00Introduction to netsplits
- 2:10Problem of split brain
- 5:02Detecting netsplit types
- 8:19Finding largest connected DCs
- 12:40Fencing safety checks
- 15:09Applying fencing solution
- 16:34Lifting the fence
- 19:48Acknowledgments and Q&A
related talks
32mStretch Clusters in Ceph: Algorithms, Use Cases, and Improvements
25mSupporting 3 Availability Zones Stretch Cluster
25mExplicitly Supporting Stretch Clusters in Ceph
31mAKKA CLUSTER MANAGEMENT AND SPLIT BRAIN RESOLUTION
35mGoing Multi-Node
32mBreaking Boundaries: Multi-cluster with Cilium cluster mesh
35mDistributed Vertex Cover Reconfiguration
06mNobody Knows What PGs are Good For, Only I Do