From On-Premise to Cloud
summary
A team migrated a legacy on-prem Hadoop setup running about 10 daily Spark batch jobs (HDFS, Hive metastore, Zeppelin) to AWS, evaluating S3+Glue against EMR. They found Glue requires wrapper scripts to pass job arguments and has ~10-minute startup per job (causing long total pipeline time), while EMR—though needing more Airflow orchestration—lets them manage cluster steps, pay only for worker nodes, and proved operationally more efficient for this workload.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:43Why we want to change
- 5:12Migration to the cloud
- 6:14What is Glue?
- 7:00HOW TO GLUE: Launch Script
- 9:42What is EMR?
related talks
38mLessons Learned from the Migration to Apache Airflow
39mWhat Can Go Wrong With Testing an ETL Pipeline?
32mOrchestrating Your Data Pipelines with Apache Airflow
44mProcessing large data with Apache Spark
19mDemystifying Modern Data Pipeline Architecture
16mBeyond ETL: Revolutionizing Data Engineering Workflows
27mModern ETL-ing with Python and Airflow (and Spark)
29mData Acquisition: Overcoming Challenges