Large Scale Batch Processing with Argo Work...
summary
Intuit runs Argo Workflows + Argo Events to manage 40,000 pipelines, with about 10,000 concurrent runs per day by moving to a multi-cluster setup (three orchestration clusters) and adding HA workflow controllers, pod disruption budgets, rate limits and semaphores. Events move from per-workflow sources to Kafka then JetStream with a sharded event source, yielding exactly-once triggering and a reported 99% reduction in event-source pods; the platform is production-proven.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:38Overview
- 2:38Intuit Data Processing Flow
- 5:15Batch Processing Platform Architecture
- 7:14Intuit Batch Processing Flow
- 9:49Scaling Argo Events
- 13:32Scaling Argo Workflow in Multi Cluster
- 15:40High Availability
- 16:25Pod Distribution Budget (PDB)
- 17:10Rate Limit for Concurrent POD
- 18:48Creating Data Processor
- 19:48Creating Data Pipeline
- 20:05Scheduling & Managing Data Pipeline
- 21:16A Cluster View of Batch Dependencies
- 21:54What we are working on Next
related talks
29mBuilding a Batch Processing Platform...
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
20mDemystifying Argo Workflows: An Architectural Deep Dive
27mLeveraging Argo Workflows To Move, Migrate and Manage All of Your Workflows
27mBuilding a Data Science Platform with Argo Workflows
06mComparing Argo Workflows and Apache Airflow at Scale
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...