Organizing data engineering around Kubeflow
summary
Kubeflow on Kubernetes runs reproducible, shareable ML pipelines where a Python function becomes a component with one-line Python-to-component conversion and components run as containers across on‑prem, cloud or hybrid clusters. Pipeline runs produce artifacts like ROC and confusion matrix and stream Kubernetes logs; experiments support clone and side‑by‑side comparison. Multi‑team workflow uses containerized components versioned by tag and CI/CD to push fixes; demo shows a failed preprocessing run fixed by updating an image and rerunning, proving end‑to‑end reproducibility and collaboration.
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 3:26Machine learning production challenges
- 6:17Kubernetes and Kubeflow overview
- 8:56Kubeflow pipelines benefits
- 12:34Pipeline execution and experiments
- 16:50Best practices for components
- 20:19Organizing teams with Kubeflow
- 23:10Demo setup and notebooks
- 27:47Running and comparing pipeline runs
- 31:43Collaboration in Kubeflow
- 32:47Fixing and iterating pipelines
- 34:20Q&A and closing
related talks
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
29mML Engineering Kubeflow
54mScaling Machine Learning Pipelines in Cloud
14mKubeflow: Where ML Meets Kubernetes
36mMachine learning pipeline Kubeflow, let's start
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
51mMachine Learning using Kubeflow and Kubernetes
31mKubeflow - Data Science on Steroids