Scaling Time Series Analysis With Argo Workflows: Patterns and Practi...
summary
Argues that Argo Workflows turns heavy time series work into scalable Kubernetes pipelines by using data partitioning and shared volumes to avoid loading whole datasets into memory, and by running DAG-based parallelism for feature extraction, model training and forecasting. Provides a GitHub repo demo (QR codes) with a dynamic DAG that runs independent partitions in parallel (example runs two keywords concurrently) and claims these patterns handle petabyte-scale datasets to cut memory bottlenecks and speed end-to-end processing.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Time series analysis overview
- 3:08Benefits of time series analysis
- 3:50Challenges in time series analysis
- 4:12Argo Workflows introduction
- 4:33DAGs in Argo
- 5:13Memory usage with large datasets
- 5:55Data partitioning with Argo
- 6:36Performance limitations in analysis
- 6:58Parallel processing with Argo
- 7:39Challenge three: Data visualization complexity
- 8:00DAGs for causal relationships
- 8:43Demo and resources
- 9:04Recap and benefits
- 9:46Q&A
related talks
29mLarge Scale Batch Processing with Argo Work...
27mBuilding a Data Science Platform with Argo Workflows
25mPanel: Argo for ML: Achieving Scalability and User Experience
06mComparing Argo Workflows and Apache Airflow at Scale
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
25mLeveraging Argo Workflows and Argo Events for Cloud Agnostic Enter...
07mArgo in an AI-Native World: Effective In-Cluster Machine Learning and A...
26mComparing Argo Workflows and Airflow in a Distributed Environment