Efficient Data Parallel Distributed Training with Flyte, Spark & Horovod
summary
Flyte orchestrates Spark or an MPI operator to run Horovod ring allreduce across ephemeral Kubernetes clusters, using the Spark operator to spawn driver+executors or MPI jobs when Spark is overkill. Demo trains Rossmann and MNIST models, achieves 98.68% accuracy on MNIST, uses ring allreduce via Horovod, and uploads artifacts to Flyte storage. Caching and memoization avoided runtime for repeated runs, and workflow recovery resumes after resource failures so distributed training actually completes.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:45Agenda
- 1:40Recap
- 3:37Overview
- 5:14Averages
- 9:18Spark
- 11:13What is Flyte
- 12:24Workflows
- 13:37User Journey
- 15:01Code Example
- 17:30Registration
- 18:30Launching an execution
- 19:21Graph of execution
- 19:39Stack trace
- 20:09Flyte concepts
- 23:33Flyte architecture
- 27:30Demo
- 35:12Example Scenario
related talks
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
29mDistributed Deep Learning with Horovod
27mDistributed Deep Learning with Horovod on Ray
25mMLOps with Flyte
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
39mFlyte: Production-Grade Data and Machine Learning Orchestration