Optimizing Machine Learning Training Infrastructure: A Governance Approach
summary
Meta describes a distributed hierarchical policy framework that enforces training workload governance via distributed hierarchical policy trees implemented as UDF nodes, with centralized policy logic and lightweight clients at schedulers, notebooks and orchestrators. It uses attribution (model type, project, stage), capacity pools and metrics like GPU hours, samples/sec and SM utilization, plus shadow-rollouts, fail‑open behavior and kill switches. Results include 100% model-type attribution coverage in Monetization, fewer wasted GPU hours, better prioritization under scarcity and fewer jobs failing late in the pipeline.
AI-written summary · about
chapters (8)
- 0:00ML training infrastructure intro
- 1:58ML training capacity challenges
- 7:14Governance dimensions
- 11:12Distributed governance design
- 21:52Meta’s implementation results
- 24:59Governance ROI benefits
- 29:31Engineers should lead governance
- 30:34Summary and call to action
related talks
21mOperationalizing ML Training Infra at Meta Scale
42mImproving Machine Learning Development Reliability
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
32mPolicy-based Compute Orchestration for AI/ML Workloads
51mKeynote: Fault Tolerant Machine Learning Operations
20mDecentralizing Compute for Machine Learning
16mMAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale
40mOptimizing AI/ML Workloads