Enhancing Runtime Reliability in LLM Training via Fine-Grained Observability
summary
Large-scale LLM training runs on tens of thousands of GPUs for days or weeks and is fragile to tiny faults. The team builds Minder, an unsupervised VAE+LSTM metric detector that reduces detection time by 99% and reports ~90% precision across CPU/GPU faults, and a lightweight CCL tracer called Minecraft that instruments trunk and flow state, localizes faults within seconds and runs on one 64‑CPU host using ~512 MB shared memory to detect NIC, compute, and communication stalls before jobs time out.
AI-written summary · about
chapters (9)
- 0:00Reliability in LLM training
- 1:05Limitations of current tools
- 2:04Minder design and insights
- 5:48Minder deployment and results
- 8:45Challenges of gray failures
- 11:34Communication dependencies
- 14:15Mycroft tracing system
- 17:15Mycroft evaluation
- 19:29Conclusion and future outlook
related talks
14mMinder: Faulty Machine Detection for Large-scale Distributed Model Training
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
23mObservability for LLMs: Understanding What’s Happening Under the Hood
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
31mReliable Data for Large ML Models: Principles and Practices
19mNo GPU Left Behind: Scaling Online LLM Training With...
32mEnd-to-End Observability for LLM Inference: From Token To GPU
39mAccelerating Open-Source RL and Agentic Inference with vLLM