Multi-branch Pipeline with Argo Workflows and CI/CD Debugging
summary
Implements a repo-level dot-workflow folder that Argo Events sensors convert into a Workflow CRD so each branch keeps its own pipeline config, using a second "cedar" workflow to merge branch files into a workflow template and submit it; debugging a sensor marshaling bug with Rookout leads to a tiny OSS fix (swap of byte-escaping function plus tests) and enables templating whole blocks. The result is an open-source tool called Piper (GitHub-only for now) that runs in-cluster, works end-to-end, and reduces developer reverse-engineering despite a ~10s orchestration overhead.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:33Shift Left
- 1:58CI/CD Consolidation
- 2:58DevOps Are DevEx Product Managers
- 4:31Multi Branch Is Very Common
- 5:32Pipeline Configuration - API
- 7:37The Solution - Argo Events
- 8:20The Solution - Seeder
- 9:28Debugging Process
- 10:21Unmarshalling Bug
- 11:01Didn't Work Either
- 11:43Second Solution
- 12:10How It Works - Seeder
- 12:59Source Code Repo
- 13:27Seeder Templating
- 14:37How It Looks
- 15:31Piper - Open Source
- 16:00Piper - Features
related talks
24mMulti-Branch Pipeline with Argo Workflows and CI/CD Debugging.
12mLightning Talk: CI/CD for Data Pipelines with Argo Workflows
27mLeveraging Argo Workflows To Move, Migrate and Manage All of Your Workflows
18mGitOps powered CICD-Glueing it together with Argo Workflows, Events
20mDemystifying Argo Workflows: An Architectural Deep Dive
23mGitOps Meets AIOps: Building Reliable AI Pipelines With Argo
23mDemystifying Argo Events: An Architectural Deep Dive
25mArgo Workflows in Action: Rabobank's Journey to Cloud Native Banking