AI Hardware Reliability at Scale
summary
At Meta over 66% of training job interruptions come from hardware failures and silent data corruptions occur at roughly one in a thousand devices versus cosmic-ray rates of one in a million. Detection scales with three concrete techniques: pit stop testing finds 70–80% of SDCs over ~45 days, ripple runs millisecond in-production tests to cover the fleet in seven days, and hardware sentiment analytics is about 41% more effective than test-based methods. SDCs corrupt gradients, create NaNs in training, and trigger subtle inference failures; Meta builds MTIA custom silicon and a factory-to-fleet pipeline to mitigate them.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 0:33Why AI Hardware Reliability Matters
- 5:00Silent Errors
- 13:15Hardware Sentiment
- 16:55Summary
related talks
21mOperationalizing ML Training Infra at Meta Scale
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
21mHow Meta Keeps its Large-scale Infrastructure Hardware Up and Running
22mAI Observability At Meta Scale
42mImproving Machine Learning Development Reliability
36mTransformers in SRE Land: Evolving to Manage AI Infrastructure
25mAI Reliability in Security: A New Data Science Paradigm