FLARE: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus...
summary
FLARE integrates a lightweight tracing daemon that instruments Python APIs and GPU kernels without backend patches, and a diagnostic engine that performs automated regression diagnosis across the full stack. Deployed on 6,000 GPUs, it cuts blind search debugging from ~50+ minutes to under five, records about 1.5 MB logs per GPU (vs PyTorch’s multi‑GB profiler), and adds ~0.43% runtime overhead. In one week of 113 jobs FLARE flags 11 regressions, with manual checks confirming most as real.
AI-written summary · about
chapters (14)
- 0:00FLARE introduction and motivation
- 1:03Why build FLARE for LLM training
- 3:14Detecting silent slowdowns
- 6:04FLARE tracing and diagnostic engine
- 7:34Tracing Python APIs without backend changes
- 9:01Measuring GPU kernel performance
- 10:26Reconstructing full stack traces
- 10:46Detecting communication hangs
- 12:39Five key metrics for regression diagnosis
- 13:46Issue latency distribution
- 14:50Void percentage metric
- 15:11FLARE overhead evaluation
- 15:53Regression detection results
- 16:36Conclusion and deployment
related talks
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
17mExtending Flamegraphs for Multi-Dimensional Performance Analysis
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
17mCharacterization of Large Language Model Development in the Datacenter
18mKeynote