Network Observability for AI/HPC Training Workflows
summary
ROCET associates jobs to RDMA hardware counters (out‑of‑sequence errors, NIC flap, local ACK timeouts) and correlates those counters with training throughput to pinpoint network‑caused slowdowns and congestion via pulse duration and contraction notifications. PARAM benchmark records and replays PyTorch execution trees using Python process group APIs to reproduce NCCL/UCC/MPI collectives in order and measure per‑collective latency and bandwidth. Chakra ecosystem supplies a unified graph schema, converters, visualizers and trace synthesis to share and simulate workloads without revealing proprietary model details.
AI-written summary · about
chapters (5)
- 0:00Network observability intro
- 2:13Observing RDMA with Rocket
- 8:27Collective comms with Prime
- 14:09Scaling benchmarks with Chakra
- 17:06Summary and future
related talks
19mAI/ML Networking Challenges: The Fast and the Finicky!
36mNetwork Machinery: A United-Front for Network Troubleshooting with CRDs
31mLive Panel: Scale-Out Networking for AI Supercomputers
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
22mMetaRoCE: Meta’s RDMA Transport
20mTurbocharging AI/ML workloads: Revving Up Speed and Resilience
01mThe Future of AI Networking Starts Here
25mResilience for AI Workloads at Scale: The Fast and the Finicky!