Perception Data Pipeline Reconstruction With Argo Workflows
summary
An end-to-end perception pipeline built on Argo Workflows and Argo Events orchestrates over 40,000 tasks daily and claims a 2x task execution speedup. The platform implements stage-to-stage streaming to avoid intermediate collects, cache warmups (Fluid) to avoid 2 TB simultaneous pulls, GPU sharing plus model quantization to raise GPU utilization ~40%, and hybrid scheduling to decouple CPU/GPU components. The system runs large-scale preprocessing, simulation and offline inference with higher throughput and lower operational cost.
AI-written summary · about
chapters (15)
- 0:00Argo Workflows intro
- 1:29Autonomous driving pipelines
- 4:00Pipeline scaling challenges
- 5:03Low data performance
- 7:11Storage bottlenecks
- 7:53GPU sharing optimization
- 8:58Heterogeneous workloads
- 10:45Multi-tenant resources
- 12:57Argo Workflows scenarios
- 13:41Concurrency control
- 15:06Event-driven pipelines
- 16:30Hybrid scheduling
- 18:38Fluid cache storage
- 20:32System architecture
- 20:53Results and outlook
related talks
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...
25mPanel: Argo for ML: Achieving Scalability and User Experience
07mArgo in an AI-Native World: Effective In-Cluster Machine Learning and A...
25mLeveraging Argo Workflows and Argo Events for Cloud Agnostic Enter...
36mArgo Workflows: Intro, Update and Deep Dive
25mHow to Multiply the Power of Argo Projects By Using Them Together
26mLeveraging Argo Ecosystem for Scalable Cloud-Native Solutions at Databricks