EROICA: Online Performance Troubleshooting for Large-scale Model Training
summary
EROICA is an online profiler that triggers only on iteration slowdowns, profiles all workers simultaneously, extracts beta, mu, sigma fingerprints for functions on the critical path (>1% time), and reduces raw data by 100,000× data reduction so a single 8‑core server can cluster fingerprints and flag outliers with Manhattan distance and preset ranges. Deployed on ~100,000 GPUs for 1.5 years, it diagnoses hardware and software faults (including a 3,400‑GPU job that yielded a 34.4% speedup) and reports 97.5% success.
AI-written summary · about
chapters (10)
- 0:00EROICA introduction
- 1:01Online monitoring limits
- 2:55Online profiling challenges
- 3:59Behavior fingerprinting
- 6:13EROICA system overview
- 7:38Fingerprint extraction
- 9:04Defining behavior fingerprints
- 11:11Localizing abnormal behaviors
- 13:22EROICA deployment impact
- 16:20Conclusion
related talks
22mAI Observability At Meta Scale
17mCharacterization of Large Language Model Development in the Datacenter
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
45mHigh-Performance and Scalable Middleware for HPC and Deep...
13mAccelerating Model Training on Ascend Chips: An Industrial System for Profiling...
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
16mReducing Operational Barriers to Model Training
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective