Lightning Talk: Step-Aligned Telemetry for Distributed PyTorch Training (Time...
summary
Step-aligned telemetry library TraceML captures per-step timing and memory, not time-sampled averages, by wrapping the loop with a trace_step context manager and running under torchrun with an aggregator. Demonstrations show a single-GPU fine-tune where injected retention causes observable memory creep across steps, and a 4-GPU run where an injected sleep on one rank produces a clear worst-rank vs median-rank skew that pinpoints a dataloader straggler. Integrations exist for Hugging Face Trainer and PyTorch Lightning.
AI-written summary · about
chapters (5)
- 0:00Step-aligned telemetry intro
- 1:45Failure modes missed
- 3:12Demo: memory creep
- 6:27Demo: rank stragglers
- 9:41Integration with frameworks
related talks
11mPer-iteration PyTorch Visibility...
21mLightning Talk: Distributed Tracing Without the Pain!
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
09mGraduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Appl...
14mLightning Talk: Cognitive and SelfAdaptive System for Distributed Tracing
14mLightning Talk: Large-Scale Distributed Training with Dynamo and...- 12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
20mTaming AI Infrastructure Failures with Agentic Debugging