Zero-cost I/O and fault tolerance in distributed deep learning
summary
Describes how Yandex built a zero-cost infrastructure for distributed deep learning on top of a distributed transactional file system, enabling distributed training without modifying single-threaded, single-process Python training code. Covers the I/O design and fault-tolerance mechanisms that let existing Python training jobs run in a distributed environment transparently.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 4:54Single-process optimizations
- 12:01State preservation
- 25:12Distributed training
- 32:13Summary and Q&A
related talks
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
49mDistributed Deep Learning
34mDemocratizing Machine Learning on Kubernetes [I]
15mDorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and
12mLightning Talk: Distributed AI Without the Infrastructure Tax
16m[Opening Keynote] Making Distributed Computing Easy
08mMaximizing ROI Through Distributed Computing and Multi-Cloud
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud