Lightning Talk: Debugging the Undebuggable: Introducing Torch.distributed.debug
summary
A new PyTorch Distributed Debug Server called torch.distributed.debug runs an HTTP front end plus lightweight worker handlers to fetch C++ and Python stack traces, native traces via PySpy, and Perfetto profiles, and aggregates an always-on flight recorder ring buffer so the UI shows which collective is stuck and which rank is slow. It uses a C++ background thread per rank, an analyzer subprocess, supports periodic or on-error dumps, is extensible via HTTP handlers, and scales to 10,000 ranks with effectively zero overhead until used.
AI-written summary · about
chapters (6)
- 0:00Torch.distributed.debug intro
- 1:24Debug server core features
- 3:28Live debug UI demo
- 6:14Debug server internals
- 8:40Extensibility and future plans
- 10:01Closing and feedback
related talks
28mArt of Debugging
52mApplication-Level Debugging and Profiling: Gaps in the Tool Ecosystem
01h 00mDebug like a pro on Kubernetes
26mA Year of Observability Across the Toolstack: Breakthroughs, Challenges, Improveme...
46mTroubleshooting Distributed Workloads ...
36mIntroducing a New Performance Monitoring Daemon
33mDebugging & Profiling Spinnaker Applications Live!
13mKeynote: Debugging in Prod Without (Lots Of) Pain