Holmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
summary
Holmes localizes slow or spiky iterations in multi‑thousand GPU LLM training by logging only communication operators, classifying abnormal operators with a random forest that updates decision thresholds for distribution drift, and traversing a communication operator graph to trace waits back to the true device. Large‑scale simulation reports 97.21% localization accuracy, a production prototype finds faults within 30.3 seconds (6.52× faster than centralized analysis) and cuts log transmission by 96%, showing the system reliably finds silent irregularities. communication operator logger adaptive random forest thresholds
AI-written summary · about
chapters (7)
- 0:00Holmes system for LLM training
- 1:50Irregularity in mega-scale GPU training
- 4:41Challenges in automatic localization
- 6:52Detecting abnormal communication operators
- 9:40Localization algorithm for root causes
- 12:57Evaluating Holmes with real clusters
- 15:27Conclusion and future work
related talks
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
19mNo GPU Left Behind: Scaling Online LLM Training With...
23mObservability for LLMs: Understanding What’s Happening Under the Hood
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3