Scaling Argo Events for Enterprise Scheduling: A Case Study fr...
summary
Intuit replaces the default Eventbus with a Kafka-based Eventbus and adds an external persistent store, separates trigger definitions into decoupled sensor metadata, and centralizes sensors into a pooled runtime. Introducing an identifier field moves filtering off the Event Source. The system processes 15,000 pipelines (~25,000 dependencies) at 150 TPS using only 20 sensor pods, reports ~85% faster filtering and claims up to 90% resource reduction compared with the original architecture.
AI-written summary · about
chapters (9)
- 0:00Argo Events scaling intro
- 1:26Platform use of Argo Events
- 3:54Scaling to 10k pipelines
- 5:02Architecture for scalability
- 8:34Optimizing Event Source Sensor
- 13:55Metadata caching
- 15:20Live demo 15k pipelines
- 17:12Production impact
- 18:39Q&A
related talks
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
03mArgo Events Update
23mDemystifying Argo Events: An Architectural Deep Dive
26mLeveraging Argo Ecosystem for Scalable Cloud-Native Solutions at Databricks
02mProject Updates: Argo Events
27mPlatform Engineering with the Argo Ecosystem: The Elastic Story
23mPerception Data Pipeline Reconstruction With Argo Workflows
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows