Network Communication Debuggability and Observability at Scale
summary
Meta instruments its collective library with a custom nickx communication library that adds real-time performance tracing at per-work-request granularity, aggregates bandwidth per rank pair, and correlates exposed collectives with token throughput. At thousands to tens of thousands of GPUs this uncovers slow ranks (PCI and switch-induced) that cost ~13% of step time or trigger 20+ minute cluster restarts, and it powers semi-online failure localization that reduces triage from hours to minutes.
AI-written summary · about
chapters (11)
- 0:00Network debuggability challenges
- 3:37LLM training scaling issues
- 4:41Training regression causes
- 8:11Real-time network tracing design
- 9:13Detecting slow ranks
- 10:38Network slowness and model degradation
- 12:45Training failures at scale
- 15:20Failure localization system
- 16:45Hardware failure analysis
- 19:18Software stalls in training
- 20:45Key takeaways and future
related talks
01h 00mSystems @Scale Summer Q&A Week 4
22mAI Observability At Meta Scale
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
32mActive Fault Finding in Networks
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
21mAnatomy of @scale Machine Learning Network Operations
22mDebugging and Observability for Distributed Ray Applications
19mAI/ML Networking Challenges: The Fast and the Finicky!