Data Gone in 60 Seconds: A Serverless ETL Heist
summary
Serverless pipeline claims to move data end-to-end in about 60 seconds and the demo proves six records traverse it live. Architecture uses API Gateway → Lambda → S3 for ingestion, enriches and queries with Athena + Glue metadata instead of a persistent database, batches and throttles with SQS batching up to 10 messages and posts to an external endpoint implemented on val.town with a SQLite backend. Deployment is defined via SAM, three Lambdas, and runs in production during the demo.
AI-written summary · about
chapters (8)
- 0:00Welcome
- 0:44Real-time data pipeline
- 2:52Serverless like car thieves
- 6:24Building ETL crew
- 11:25Meet Paul Chin Jr
- 13:11Live demo
- 17:34Architecture breakdown
- 22:33Key takeaways
related talks
34mDesigning Data Intensive Applications In Serverless Architecture
25mA simple serverless data pipeline
42mBuild and automate a modern serverless data lake on
41mTechnology mix: Around the serverless in 180 days
41mUnlocking Faster & Efficient Data Processing w/ Serverless
31mAccelerate business innovation with serverless applications
49mServerless Architectural Patterns and Best Practices
01h 00mServerless Architecture - Tales from a world without servers