Designing Data Intensive Applications In Serverless Architecture
summary
Claims a serverless ETL blueprint that cut costs by 70% cheaper and scales to 10–100s GB/s using AWS Lambda, S3 data lakes, Kinesis/SNS/SQS for transport, DynamoDB (and Streams), Glue for SQL ETL, Step Functions for orchestration and Neptune for graph queries. Implements a 3+ Lambda minimal pipeline, hides DB access behind an API-lambda to reduce connections, decouples writers so retries are per-writer, and uses dead‑letter + reprocess loop; notes the 15 minute Lambda limit and real deployments for 300+ customers.
AI-written summary · about
chapters (16)
- 0:00Introduction and speaker intro
- 1:03Serverless cost and popularity
- 2:06Data intensive applications overview
- 3:29AWS Lambda for data pipelines
- 4:57Data extraction from data lakes
- 6:21Transport options comparison
- 8:33Batch vs real-time processing
- 9:58Data transformation best practices
- 12:54Reliability and error handling
- 17:50Data loading strategies
- 21:52Consumer-focused data design
- 23:16Job completion tracking
- 25:25Data analysis pipeline
- 26:34Scalability in serverless
- 28:44Design mistakes and solutions
- 29:31Q&A session
related talks
49mServerless Architectural Patterns and Best Practices
23mSimplifying design complexities using Serverless Architecture on AWS
25mData Gone in 60 Seconds: A Serverless ETL Heist
01h 00mProcessing a Data Lake on Node.js in a Serverless Architecture
42mBuild and automate a modern serverless data lake on
41mUnlocking Faster & Efficient Data Processing w/ Serverless
30mEvolving Serverless Architectures
31mAccelerate business innovation with serverless applications