Lightning Talk: CI/CD for Data Pipelines with Argo Workflows
summary
Implements CI/CD for data pipelines by storing Argo Workflows WorkflowTemplates in git, triggering builds with Argo Events, and running PR-time tests against sample inputs. Demo applies a "doubler" template, runs three assertions, and a breaking change (multiply → add) produces two failures and one success, proving tests catch regressions. Uses a workflow-of-workflows pattern to apply templates then execute validations; proposes semantic-versioning by name suffixes or labels.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:24GOALS FOR THE TALK
- 1:22BACKGROUND: CI/CD FOR DATA PIPELINES
- 3:35BACKGROUND: WORKFLOW TEMPLATES
- 4:40TESTING WORKFLOW TEMPLATES
- 5:11VERSIONING WORKFLOW TEMPLATES
- 7:08ARGO WORKFLOWS
related talks
12m⚡ Lightning Talk: On-Premise and SaaS CI/CD Large-Scale Production Automation with Argo Services
22mLeveraging Argo WorkflowTemplates Within a Data Platform
25mHow to Multiply the Power of Argo Projects By Using Them Together
12mLightning Talk: Migrating 1,300 Jenkins Jobs To Argo Workflows: Lessons From a…
36mArgo Workflows: Intro, Update and Deep Dive
24mMulti-Branch Pipeline with Argo Workflows and CI/CD Debugging.
39mBuild and Deploy Your Application with Argo on Kubernetes
52mCloud-Native Kubernetes Workflows on AKS with Argo