Building Data Lake Platform Fully Serverless
summary
Builds a self-service data lake for Polestar cars using only AWS serverless pieces — Kinesis Firehose for ingestion, Lambda for processing, S3 as object store, DynamoDB for temp state, EventBridge for routing, Glue/Athena and Lake Formation for catalog and access. It centralizes data from ~60 internal products and many partners, solves Kinesis 256 KB limits with signed‑URL S3 uploads and metadata folders, reduces cost with crawler cadence, S3 lifecycle and TTLs, and reaches hundreds of GBs (approaching ~1 TB) in production so far as a fully serverless solution.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:34Problem statement
- 10:38Choosing data lake
- 16:00Core architecture
- 20:21Landing zone design
- 28:51Processing and storage
- 33:23Self-service platform
- 36:48Key takeaways and solution architecture insights
- 39:59Cost optimization
- 42:07Scaling challenges
- 44:35Alternatives comparison
- 46:20Data lake vs big data
related talks
47mBuilding Data Lake Platform Fully Serverless
42mBuild and automate a modern serverless data lake on
37mBuild a Modern Data Platform on AWS
34mDesigning Data Intensive Applications In Serverless Architecture
01h 00mProcessing a Data Lake on Node.js in a Serverless Architecture
32mBuilding and scaling a log analytics platform - a serverless approach
49mServerless Architectural Patterns and Best Practices
45mBuilding a Lakehouse on the Microsoft Intelligent Data Platform