Anatomy of @scale Machine Learning Network Operations
summary
Operations at Coreweave target synchronously connecting 128,000 GPUs with zero packet loss across millions of ports and tens of thousands of miles of fiber, so monitoring and testing are custom built, not off‑the‑shelf. Infrastructure uses an eventing stack and Coreweave Cumulus CLI for global, chained actions; half of network engineering are developers. NPI means repeating full telemetry, fabric, and ASIC validation on compressed timelines, and event‑driven automation reduces meantime to detection and resolution.
AI-written summary · about
chapters (12)
- 0:00AI and ML network ops intro
- 1:06Traditional vs AI/ML networks
- 3:11Four cornerstones of AI ops
- 5:41Frameworks in scaling ML
- 7:45New product integration challenges
- 9:51Org structure and developer focus
- 10:52Real-time testing and fabric validation
- 13:46Infrastructure automation and truth
- 15:35Cumulus CLI and tooling
- 17:40Vendor partnerships matter
- 19:04Engineering velocity in NPI
- 20:07Future ops robustness focus
related talks
25mHow We Power the Largest AI Deployments on the Planet: Running Vir...
17mMeta’s Network Journey to Enable AI
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
23mNetwork Communication Debuggability and Observability at Scale
23mNetworking for GenAI Training and Inference Clusters
18mNetwork Observability for AI/HPC Training Workflows
21mDC Networks for Generative AI
26mPetabyte-Scale Video Curation on CoreWeave