Deep Learning at Scale
summary
Pipeline builds image ingestion with Apache Spark -> Petastorm -> Parquet on distributed storage, trains Keras/tensor flow models with Horovod for data‑parallel GPU scaling, and tracks runs with MLflow while exploring hyperparameters via Hyperopt. Claim scales to “couple of millions” of images; small benchmark on chest X‑rays (≈110 images) uses DenseNet and shows about 100 minutes on 4 GPUs versus ~25 minutes on 16 GPUs. Demonstrates working end‑to‑end distributed training, gradient averaging, and reproducible hyperparameter search.
AI-written summary · about
chapters (6)
related talks
49mDistributed Deep Learning
23mRethinking scalable machine learning with Spark ecosystem
42mDeep learning beyond the learning
34mDemocratizing Machine Learning on Kubernetes [I]
13mFrom Silicon to Scale: Training at Scale
37mDistributed Deep Learning on Apache Mesos with GPUs and Gang Scheduling
18mDeep learning for Space imagery applications
43mDeep learning beyond the learning