Embrace DevOps Practices to ML Pipeline Development
summary
Built a compiler from KFP SDK to Tekton that runs pipelines on Tekton while injecting artifact annotations + post‑processing to push artifacts into KFP’s MLMD so the KFP UI shows DAGs and metadata although execution uses Tekton. CI/CD uses IBM Toolchain + Tekton task catalog for nightly end‑to‑end runs, Slack alerts and DevOps Insights, with deployments to a pre‑configured Kubernetes cluster. Result: end‑to‑end nightly verification and automated releases work for simple pipelines; complex KFP DSL features don’t map cleanly so some workflows remain hard to express.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 2:31Kubeflow and Tekton integration
- 6:02Artifact tracking solutions
- 10:21DevOps practices for CI/CD
- 13:27Toolchain and implementation
- 20:11Challenges and future outlook
related talks
42mAI Pipelines Workflows and ML lineage using Tekton Pipelines
06mMLOps, Kubeflow, and Tekton
41mDogfooding Tekton project with Tekton
29mCloud Native CD Pipelines with Tekton
47mTekton, a new approach to pipelines
31mBringing ML Workflows to Heterogeneous Cloud Native Machine Learning Pla...
43mTekton From source to production inside Kubernetes
34mFrom Code to Deployment: A Deep Dive into Pipelines as Code