Taming AI Infrastructure Failures with Agentic Debugging
summary
NCCL watchdog timeouts cause massive pain in distributed PyTorch training — at ~2,000 GPUs job success dips below 2% — and Meta finds >99% of timeouts are scheduling mismatches not NCCL slowness. The team uses the PyTorch flight recorder to collect per-rank ring-buffer traces, builds visualizations to spot the first mismatching collective, and automates a runbook plus static checks. Converting that flow to agentic debugging drops triage from hours or days to minutes; Monarch’s centralized controller aims to eliminate these mismatches entirely.
AI-written summary · about
chapters (9)
- 0:00AI reliability challenges
- 2:55NCCL watchdog timeouts
- 5:26Execution flow mismatches
- 8:02PyTorch flight recorder
- 11:31Scheduling mismatch causes
- 13:36Manual debugging limits
- 14:37Agentic debugging for triage
- 18:33PyTorch innovations outlook
- 19:15Agentic debugging future
related talks
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
21mOperationalizing ML Training Infra at Meta Scale
20mBringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Tra...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
22mGenAI Training In Production: Software, Hardware & Network Considerations
28mScaling AI Infrastructure
21mAI Hardware Reliability at Scale
31mDashboards & Dragons: Crafting SLOs To Tame the AI Platform Cha...