Batch Processing Tekton Pipelines in OpenShift
summary
Tekton runs batch pipelines on OpenShift to process 250,000 archives/day into a data lake with >100 tables and ~110 million rows daily; it matches Argo Workflows on performance and has strong CLI/UI observability. Limits: no native loop support, so a Python pipeline model plus a compiler expands iterations into static YAML; default one-hour timeouts and inability to parametrize memory per reusable task require per-pipeline task definitions; garbage-collection gaps force a custom garbage-collector script. Overall works with these workarounds and a proposed boilerplate.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:23Data Processing
- 3:13Questions
- 3:50Mandatory Agenda
- 4:46Types of Pipelines
- 7:43Pipeline Managers
- 9:02Technology
- 11:20Open Source
- 13:44Daily Pipeline
- 14:28Experiments
- 17:41Best Practices
- 22:13Loop Support
- 24:54Backfields
- 29:45memory limits
- 30:58daytoday pipeline
- 33:29next steps
- 34:18QA
related talks
21mHow to Build CI/CD Pipelines for a Microservices Applicatio
39mPower-Up With Clouds And Pipelines
47mTekton, a new approach to pipelines
30mSave Your Pipelines (and Clusters!) with Tekton Results
22mLessons Learned Evolving K8s Extensions: The Future Of Tekton Pipelines
10mWorkflows - Quick, Easy CI with Tekton
29mBuilding a Batch Processing Platform...
41mDogfooding Tekton project with Tekton