Mid-air airplane repair: Troubleshooting
summary
WhatsApp runs a live cluster serving 1.5B users across roughly 10,000 nodes and debugs on production without disrupting clients. Tools shown include an internal ECTL/inspection shell, a PowerShell extension (GitHub) to call private OTP functions, and microstate accounting plus gdb for nonblocking VM introspection. Case studies: a process queue of ~112,300 items inspected live, a prototype freed 230 MB from one process, and changing an allocator cut RAM from ~18 GB to ~3 GB; fixes roll out node→rack→DC with config/app/OTP tiers.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:25Workflow
- 3:36Detection
- 5:46Confirm
- 7:07Analyze
- 13:54Monitor
- 14:41Computing resources
- 17:33Memory introspection
- 19:42Debugging
- 21:40Prototyping
- 23:54Mitigation
- 26:21Documentation
- 33:44Microstate Accounting
- 38:44Crash Dump Analysis
related talks
30mAbove the Clouds: Investigating Systems from 32,000 ft.
34mChaos Debugging: Finding and Fixing Microservice Weak Points
24mDebuggin a Large Node Cluster
25mDebugging at Scale in Production with kubectl debug
39m30 minutes of Failure
39mHow to Use Common Investigation Practices in Troubleshooting
53mDebugging Go Applications
19mLive Q&A