Anatomy of @Scale Machine Learning Network Operations
summary
Claims that AI/ML training must run at line rate across 128,000 GPUs with zero flaps, so CoreWeave builds blueprints around four cornerstones: people, frameworks, systems, NPI. Systems include a custom Master Control Program (MCP) linking ZTP, an enriched source-of-truth, streaming telemetry and a Cumulus CLI for bulk operations across many thousands of switches (64 ports each), enabling event-driven automation to manage half a million fibers and over a million optics and deliver stable, production NPI rollouts.
AI-written summary · about
chapters (13)
- 0:00AI ML network ops intro
- 1:07Problem statements in AI/ML networking
- 3:14Four cornerstones at scale
- 5:20Frameworks' role in ops
- 6:24Systems and automation at scale
- 7:48New product integration challenges
- 9:12Solving cornerstone problems
- 10:14Building SME in teams
- 11:39Observability and event automation
- 13:46Master Control Program and tooling
- 16:16Vendor partnerships in NPI
- 18:02Engineering velocity and problem solving
- 19:08Closing thoughts and outlook
related talks
17mMeta’s Network Journey to Enable AI
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
23mNetworking for GenAI Training and Inference Clusters
21mDC Networks for Generative AI
19mScaling AI Network with DSF
06mOpening Remarks