Modern Data Orchestrators
summary
Practical checklist for modern orchestration: use Airflow+DBT or Airflow+Polars for small fast jobs and reserve Spark for true big data, generate DAGs from config/templates and enforce CI/CD so hundreds of near‑real‑time pipelines don’t become unmanageable, integrate OpenLineage/OpenMetadata and Great Expectations/Pandera for lineage and validation, deploy compute on Kubernetes (or cloud) not inside the orchestrator, avoid vendor lock‑in by using open metadata formats — these steps are proven in enterprise refactors.
AI-written summary · about
chapters (11)
- 0:00Introduction to modern data stack
- 3:01Challenges with modern data orchestrators
- 7:42Key features of modern orchestrators
- 10:17Infrastructure and scalability considerations
- 13:32Data lineage and observability importance
- 16:45Data validation and quality assurance
- 18:10Challenges in adopting new tools
- 20:44Refactoring existing data stacks
- 22:10Best practices for data orchestration
- 24:18Integrating observability and lineage
- 25:25Conclusion and evolution advice
related talks
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
42m`New` Workflow Orchestrator in town: "Apache Airflow 2.x"
17mData Mesh: The Speedrun
22mOrchestrate your data pipelines using Apache Airflow
39mOrchestration of Big Data systems: A survivor's guide
41mModern Data Architectures – What They Are and How to Implement?
26mGo with the Flow: Automating Your Workflows with Airflow
56mWhy Airflow is a secret ingredient in MLOps