Unleashing the serverless data pipelines
summary
Compares three architectures—Talend on Hadoop, Spark on EMR with Airflow, and AWS Glue ETL—and chooses Glue after modeling reduced input (rolled to five‑minute granularity from vendor probes down to 10 GB/day). Shows concrete numbers: a 300 GB parquet job costs about $192 on EMR versus ~$350 on Glue, but overall stack cuts total cost ~75% while throughput doubles. Builds CI/CD with boto3, adds DynamoDB logging, CloudWatch, EventBridge/Lambda/SNS alerts, Spark history servers; Glue works but has ~10‑minute job startup and lacks workflow‑level monitoring.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:42Welcome
- 1:27Domain Knowledge
- 2:45CDN Scale
- 4:10Business Use Case
- 6:28System Overlay
- 7:41Background
- 12:13Option 1 EMR
- 16:42Factors
- 18:15Latency Tolerance
- 22:05Cost
- 25:08Data connectors
- 26:36Maintenance
- 27:00Decision Making
- 27:22Benefits and Limitations
- 28:34Solution
- 29:49Conclusion
- 31:30References
- 31:45Questions
- 34:29Thank you
related talks
25mA simple serverless data pipeline
44mOptimize Costs with Serverless Data Integration Using AWS Glue
28mInnovations in AWS analytics: Data processing
34mDesigning Data Intensive Applications In Serverless Architecture
42mBuild and automate a modern serverless data lake on
13mHow to automate legacy ETL migration to AWS Glue
16mAutomated Data Analytics
05mAWS On Air ft. Amazon EMR Serverless Quick Tip 2