Building a Batch Processing Platform...
summary
Intuit builds a multi-cluster batch platform using Argo Workflows, Argo Events and Argo CD plus Kafka to run ~40–50k pipelines with thousands of concurrent executions; runtimes include spark on Kubernetes, EMR, Databricks and containers. The platform delivers automated CI/CD, namespace and IAM provisioning, low‑code pipeline authoring and dashboards, scales across four production Kubernetes clusters, and achieves a one-minute job deployment guarantee and four nines availability by pre-warming IPs, load‑balancing clusters and decoupling orchestration from runtime.
AI-written summary · about
chapters (11)
- 0:00Start
- 2:08Platform Capabilities
- 7:22Spark On Kubernetes
- 8:38Intuit Processing Flow
- 9:41Batch Platform Architecture
- 13:00Batch Processing Flow
- 15:07Platform Operational Excellence
- 18:13Developer Operational Excellence
- 18:52Creating Data Processor [ref 2]
- 19:47Creating Data Pipeline [ref 2]
- 21:05Execution Demo
related talks
29mLarge Scale Batch Processing with Argo Work...
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
33mScaling to Thousands of Data & CI/CD Pipelines Using Argo and Virtual...
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
25mMigrating CI/CD from Jenkins to Argo
23mArgo at Scale: Navigating Complex Multi-Dimensional Deployments Across H...
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...