Processing a Data Lake on Node.js in a Serverless Architecture
summary
Processing a data lake on Node.js using AWS Lambda in a serverless architecture: covers architectural cloud patterns for large-scale data processing, leveraging Node.js strengths and optimizations inside Lambda functions, and scaling to thousands of concurrent Lambda instances (up to ~7000). Discusses use of streaming, queues and databases, tuning and protocol choices for throughput, plus profiling, logging and monitoring practices for production deployment.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:08Data lake
- 7:19Node.js and serverless
- 11:10Data processing
- 17:49Dependency issues
- 23:23Error handling
- 37:57Error types
- 41:24Reading S3 data
- 47:23Scaling functions
- 51:55Architecture flexibility
- 53:58Performance and cost
- 56:04Q&A
related talks
34mDesigning Data Intensive Applications In Serverless Architecture
41mUnlocking Faster & Efficient Data Processing w/ Serverless
49mServerless Architectural Patterns and Best Practices
45mHow to organize ETL with Node.js in a serverless architecture
47mBuilding Data Lake Platform Fully Serverless
42mNo Server Is Better Than No Server: Building Blocks for Serverless Architectures
32mThe Serverless Tidal Wave
01h 00mServerless Architecture - Tales from a world without servers