End-to-end Deadlock Debugging Tools at Facebook
summary
Facebook builds four concrete tools: folly::Synchronized to force smaller critical sections and encode lock ownership in types, ThreadSanitizer run on CI and on a subset of production via shadow traffic to catch races and lock inversions, an eBPF deadlock detector that samples running binaries every ~15 minutes and reports waits‑for cycles across pthread mutexes, and a gdb extension that reads mutex internals (works on cores) to show thread-to-thread deadlocks; a real six‑mutex cycle is found in production, TSAN costs CPU/memory.
AI-written summary · about
chapters (9)
- 0:00Intro
- 2:37Debugging Deadlocks is Difficult
- 9:05Approach #2: ThreadSanitizer Testing
- 14:01eBPF Deadlock Detector: Example
- 15:55eBPF Deadlock Detector: How it Works
- 19:01Approach gdb Deadlock Extension
- 19:32gdb Deadlock Extensions Example
- 20:12gdb Deadlock Extension: How it Works
- 21:54Deadlock Debugging Tools, Pros and cons
related talks
47mDeadlocks: The Dark Side of Concurrency
01h 01mGeneric Locking in C++
01h 04mDebugging data races
26mConcurrency: Unlocking a Solution
15mSound Dynamic Deadlock Prediction in Linear Time
43mSelf-Aware Applications: Automatic Production Monitoring
26mTackling Contention: The Monsters Inside the 'sync.Locker'
01h 06mLocks are tricky — let's understand them by building one