Bringing ML Workflows to Heterogeneous Cloud Native Machine Learning Pla...
summary
KFP v2 IR provides a platform-neutral pipeline spec that lets Kubeflow Pipelines and Google Vertex AI share the same component-based representation; an abstraction layer (compiler, execution client, object API) reduces backend changes to a single package so adding a new runtime requires editing one adapter instead of UI/SDK/server everywhere. Implementations include Argo and a Tekton port that reproduces artifact/metadata features, a KFP v2 IR alpha used for Tekton parity with Argo, synchronous driver/publisher metadata enables TFX-compatible lineage, caching and faster reproducible runs; V2 Alpha 4 is available.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:58Pillars of Al Lifecycle - Datasets, Models...
- 1:14and Pipelines
- 2:57Define Pipeline with Python SDK
- 3:34Argo Workflows
- 4:51Kubeflow Pipelines with Tekton hits v1.0
- 5:42Benefits of metadata and artifact tracking
- 6:50Lineage Tracking
- 9:14TensorFlow Extended-Using MLMD as metadata store
- 9:51Kubeflow Pipelines v2 main goals
- 10:52Machine Learning Metadata in v1
- 13:09Pipeline Spec in v1
- 14:01Intermediate Representation in v2
- 15:21New Orchestration Controllers
- 17:28Components
- 18:31Abstraction Layer for Orchestration Engines
- 19:08Abstraction Layer Benefits
- 21:54Abstraction Layer Features: Execution Client
- 22:45Abstraction Layer Features: Execution Spec
- 25:31Summary
- 26:54References
- 28:58Smart Runtime
related talks
36mDemocratization of ML Pipelines and Bringing ML Workflows to Heterogeneous...
29mML Engineering Kubeflow
09mKubeflow Pipelines v2: The Next Generation of MLOps on Ku...
27mKubeflow for enabling AI Powered Drug Discovery and development...
54mScaling Machine Learning Pipelines in Cloud
44mEngaging the Kubeflow Community: Building an Enterprise-Ready AI/ML Platform
14mKubeflow: Where ML Meets Kubernetes
31mKubeflow 2026 State of the Union: Orchestrating the Full ML Lifecycle