Comparing Argo Workflows and Apache Airflow at Scale
summary
Argo uses Kubernetes CRDs with a workflow controller and Argo server, and benchmarks show Kubernetes API becomes the core limit so scale follows cluster capacity; Argo pairs well with object stores and an optional SQL archive and now has a usable Hera Python SDK. Airflow runs on a scheduler, executor, webserver and metadata DB, with scheduler and metadata DB bottlenecks unless those are tuned, and still benefits from a rich connector library but more moving parts to operate.
AI-written summary · about
related talks
26mComparing Argo Workflows and Airflow in a Distributed Environment
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
36mArgo Workflows: Intro, Update and Deep Dive
29mLarge Scale Batch Processing with Argo Work...
38mArgo and Tekton: Pushing the Boundaries of the Possible on Kubernetes
27mBuilding a Data Science Platform with Argo Workflows
52mCloud-Native Kubernetes Workflows on AKS with Argo
36mAutomating Cloud-native Spark Jobs with Argo Workflows