Where’s My Training Job? Ray & GPU Observability at Datadog
summary
Datadog builds a unified job-level view that preserves Ray job identity across cluster lifetimes, ties each rank to process, host, GPU, traces, and topology, and links failures to GPU health, utilization, and cost. Implementation uses continuous tracing, topology visualization, MCP and Bits AI to assemble causal narratives; tracing shows a CUDA sync taking 166 ms (>80% of a step) and runs with <2% overhead. At scale they observe one failure every 3 hours on a $36M/month fleet, and NVLink XID/retransmit signals localize real root causes that dashboards alone miss.
AI-written summary · about
chapters (12)
- 0:00Introduction to GPU monitoring
- 0:27Ray and distributed training challenges
- 1:52Why GPU health isn’t enough
- 3:14Finding the straggler in training
- 4:39Cost of inefficiency at scale
- 5:42Datadog’s GPU monitoring solution
- 6:44Identifying the slow rank
- 7:06Root cause analysis with traces
- 8:52Connecting traces to code
- 9:14Topology visualization for diagnosis
- 10:38Future of job-level observability
- 11:21Summary and call to action
related talks
22mDebugging and Observability for Distributed Ray Applications
22mAI Observability At Meta Scale
08mNetwork Performance Monitoring
01h 41mDatadog DASH 2019 Keynote
29mKubeflow Trainer Observability: Real-Time Progress Tracking for Reliab...
19mOops my app performance sucks... what if observability could be a game changer?
11mPer-iteration PyTorch Visibility...
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads