Day2 02 Sean Klein
summary
A production bug in async Rust turns out to be a non-local deadlock called Futurelock: multiple HTTP handlers block on a connection-pool MPSC send because the semaphore permit is acquired but never completed, observed with DTrace probes attached to Tokio, omdb, a core dump extracted a core file and inspected in Ghidra. The investigation shows Tokio select and non-atomic send steps let a half-polled future hold a permit forever, so spawning tasks or eagerly canceling partially polled futures fixes the class.
AI-written summary · about
chapters (14)
- 0:00Future lock bugs intro
- 1:39Dog food rack outage debug
- 3:05Tokio DTrace probes
- 5:56MPSC channels normal work
- 9:06Wedged MPSC state investigation
- 11:30Future lock mechanism discovery
- 13:39Future lock definition
- 15:48Root cause of future lock
- 17:54Polling management and dependencies
- 20:38Preventing future lock strategies
- 21:20Code examples and patterns
- 24:07Future lock detection difficulty
- 26:36Future lock production diagnosis
- 27:40Future lock prevention
