Unlocking Faster & Efficient Data Processing w/ Serverless
summary
Shows how to orchestrate AWS services to process massive datasets using AWS Lambda as compute and Step Functions Distributed Map as the coordinator. Demonstrates iterating S3 input, batching items, tuning concurrency up to 10,000 and using an express child workflow, writing results to DynamoDB and handling partial failures with a tolerated failure threshold. Processes 500,000 files totaling 40 GB and finishes under 2.5 minutes, proving the approach scales and works in practice.
AI-written summary · about
chapters (16)
- 0:00Serverless data processing intro
- 1:16Designing data systems
- 3:02Large-scale processing challenges
- 5:56Distributed computing for big data
- 8:23Distributed systems ops
- 9:29Serverless reduces ops burden
- 10:32AWS Lambda for compute
- 13:42Lambda replaces compute nodes
- 14:27Step Functions as coordinator
- 16:54Low-code workflow benefits
- 19:44Scheduling & event workflows
- 22:33Serverless data solutions overview
- 23:16Step Functions Map for parallelism
- 27:07Weather data processing demo
- 39:15Use cases & success stories
- 40:42Implementation resources
related talks
34mDesigning Data Intensive Applications In Serverless Architecture
01h 00mProcessing a Data Lake on Node.js in a Serverless Architecture
31mAccelerate business innovation with serverless applications
49mServerless Architectural Patterns and Best Practices
32mServerless Data Processing and Machine Learning
43mServerless in Java Lessons learnt
25mData Gone in 60 Seconds: A Serverless ETL Heist
42mNo Server Is Better Than No Server: Building Blocks for Serverless Architectures