EMR Serverless
summary
Amazon EMR Serverless launches Spark and Hive jobs without clusters, automatically provisioning workers with default worker = 4 vCPU / 16 GB / 20 GB and fine‑grained scaling that adjusts per-stage (example: 10→40→5 workers), claiming an EMR runtime 2x faster than open source. Batch starts take ~90–120s, interactive queries use a pre-initialized warm pool for sub‑second starts, jobs rerun in a healthy AZ on failure, capacity can be capped (example: 1,000 cores), and billing is by aggregated vCPU/memory/storage rounded to the second (1‑minute min).
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:27What we're going to talk about
- 3:57How it works
- 5:44Simpler to use
- 6:51No need to guess cluster sizes
- 8:23All the benefits of Amazon EMR without managing clusters
- 9:35Fine-grained scaling saves costs
- 11:205. Resiliency to Availability Zone failures
- 13:57Enable Interactive applications
- 16:31Easy to switch deployment models
- 23:37Pattern #1: Data pipelines
- 25:10Pipelines become simple to run
- 25:34Automatically scale when running stages in pipeline
- 26:12Workers are decommissioned when pipeline finishes
- 26:36Shared clusters for teams on premises using queues
- 27:00Shared clusters with Amazon EMR on Amazon EC2 using auto scaling
- 27:37Shared applications with Amazon EMR Serverless
- 28:20Per-Job execution role
- 30:15Debugging jobs is super simple
- 31:44Workers start when applications start
- 33:42Visibility and monitoring
- 34:48Create application
- 38:51Example
- 39:47What is available in preview
related talks
05mAWS On Air ft. Amazon EMR Serverless Quick Tip 2
05mAWS On Air ft. Amazon EMR Serverless Quick Tip 1
38mUnleashing the serverless data pipelines
28mInnovations in AWS analytics: Data processing
08mPivot
34mHive on Spark #bbuzz
32mMove to managed analytics
24mCICD Pipeline and delivery of Apache Spark Applications on the cloud using AWS