AI Pipelines Workflows and ML lineage using Tekton Pipelines
summary
KubeFlow Pipelines gets ported to run on Tekton instead of Argo, providing a Python DSL plus KubeFlow Pipelines on Tekton execution. Metadata and lineage use an MLMD-backed pipeline: a MLMD metadata writer watches emptyDir outputs and copies artifacts so models trace to datasets, hyperparameters and framework versions. IBM ships Watson Studio Pipelines (drag‑and‑drop UI) and a managed Tekton service, adds Tekton custom tasks for loops/recursion and an AnySequencer dependency primitive. Machine Learning Exchange (ml-exchange.org) hosts assets including Project CodeNet: 14M samples (~0.5B lines).
AI-written summary · about
chapters (4)
- 0:00Start
- 8:06Benefits of metadata logging
- 9:52Lineage Tracking
- 15:01Announcing: Machine Learning exchange (MLX)
related talks
22mEmbrace DevOps Practices to ML Pipeline Development
06mMLOps, Kubeflow, and Tekton
29mML Engineering Kubeflow
37mEnd-to-End ML Pipelines with KubeFlow and TensorFlow Extended (TFX)
36mDemocratization of ML Pipelines and Bringing ML Workflows to Heterogeneous...
31mBringing ML Workflows to Heterogeneous Cloud Native Machine Learning Pla...
05mCI/CD Pipelines and Machine Learning
39mDeploy your ML Workloads on Kubernetes with Kubeflow