From timeout to sub-second: solving scale dependent deadlocks in distributed systems
summary
A production-only deadlock in Apache HBase surfaces once clusters exceed about 50 region servers: a disaster-recovery log-roll pipeline repeatedly hits a 60-second timeout because a redundant synchronous RPC creates a circular dependency between region servers and the HMaster. Diagnosis uses synchronized thread dumps collected across many nodes to prove the blocked threads, and a small code change — removing an unnecessary namespace/table existence check during the global barrier — collapses the failing operation from 60s to a few hundred milliseconds.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:42Scale at Flipkart and HBS basics
- 2:09HBS architecture and WAL log roll
- 4:15Why backup needs log roll and checkpointing
- 6:03Problem statement and observations
- 7:49Hypotheses testing and timeouts
- 9:11Debugging with logs and thread analysis
- 11:22Deadlock in log roll procedure
- 13:08Proving deadlock with thread dumps
- 14:34Scale impact on deadlocks
- 15:38Root cause and solution
- 16:41Fix deployment and performance
- 17:03Key learnings
- 18:49Q&A
related talks
39mTrust Deterministic Execution to Scale & Simplify Your Systems
23mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
48mHow to build distributed concurrency primitives
28mAchieving Rapid Response Times in Large Online Services
29mPost Mortems for 4 Years of Remote Execution
46mFault tolerance in Distributed Systems: A Case Study in Apache Spark
46mWhat can go wrong in a distributed system - experience from...
14mHarnessing Cascading Timeouts for System Resilience