Bringing Kubeflow Training Local: SDK-Driven “Local-exec”...
summary
Kubeflow Trainer adds a local_exec mode that runs training jobs on a laptop (containerized via Docker/Podman or as local subprocesses) while keeping the same SDK API so code runs unchanged on cluster or locally. Claims a 23% ML job failure rate to justify faster iteration, demos a modified PyTorch MNIST example, and describes three backends (subprocess, Docker, Podman, kind). Notes container mode can simulate distributed jobs and enforce limits; subprocess mode lowers install friction but can’t guarantee feature parity.
AI-written summary · about
chapters (6)
- 0:00Intro
- 3:18Architecture Overview
- 8:19Implementation Options
- 10:38Benefits
- 11:42Next steps
- 12:49Questions
related talks
36mStreamlining ML Workflows With the Unified Kubeflow SDK
46mAccelerating Machine Learning DevOps with Kubeflow
51mMachine Learning using Kubeflow and Kubernetes
20mEmpowering ML Workloads With Kubeflow: JAX Distributed Training and LL...
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
31mKubeflow Intro – Michał Jastrzębski & Ala Raddaoui, Intel
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer
22mThe Containerization of Machine Learning: TensorFlow: Kubernetes, and Kubeflow