ETL at Scale: Optimizing PySpark + Airflow Workflows
summary
Uses PySpark + Airflow on Kubernetes with Terraform and Helm to make ETL fast, reliable, and costed: executor sizing ~4–8 GB, memory fractions ~0.6/0.2, broadcast joins for dims <100 MB, salting with 16 buckets to remove stragglers, checkpointing and idempotent writes, Prometheus + structured logs + OpenTelemetry and Great Expectations for observability, a reported pipeline speedup from 19m to 3m and typical 30–50% efficiency gains, operational targets like 99.9% uptime and a four‑week phased rollout plan.
AI-written summary · about
chapters (8)
- 0:00Introduction to Cloud Native ETL
- 1:40ETL Scaling Challenges
- 3:08Building a Stable Foundation
- 4:45Optimizing Spark Performance
- 7:32Airflow Orchestration Techniques
- 11:50Monitoring and Observability
- 13:08Implementation Roadmap
- 16:50Key Takeaways and Conclusion
related talks
27mModern ETL-ing with Python and Airflow (and Spark)
38mLessons Learned from the Migration to Apache Airflow
32mBuilding Analytics Workflow using Airflow and Spark
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
36mElegant data pipelining with Apache Airflow
31mIntro to GKE setup of airflow with helm and terraform
29mBuilding Large Scale ETL Pipelines with Dask