How to organize ETL with Node.js in a serverless architecture
summary
Organises large-scale ETL processing using Node.js streams in a serverless architecture, handling ndjson and Parquet data formats. Covers unusual/edge use cases for Node-based ETL and shows orchestration options using Airflow and AWS Step Functions.
AI-written summary · about
chapters (20)
- 0:00ETL processes intro
- 3:09Delivery complexity
- 6:39OSM data features
- 9:54XML to JSON conversion
- 11:19AWS Athena architecture
- 13:04Athena XML issues and fix
- 15:34Query optimization via partitioning
- 18:30Further query optimization
- 21:50Results and demo
- 22:33Test to production move
- 23:37File processing with streams
- 27:50Faster binary processing
- 30:20Processing all files
- 31:49Error handling and monitoring
- 34:20Q&A
- 34:41Production issues and fixes
- 36:55Chunking for parallel processing
- 37:40Monitoring data processing
- 39:04Metrics and notifications
- 41:15ETL architecture takeaways
related talks
01h 00mProcessing a Data Lake on Node.js in a Serverless Architecture
34mDesigning Data Intensive Applications In Serverless Architecture
41mUnlocking Faster & Efficient Data Processing w/ Serverless
25mData Gone in 60 Seconds: A Serverless ETL Heist
45mBuilding Event-driven Serverless Applications
42mBuild and automate a modern serverless data lake on
49mServerless Architectural Patterns and Best Practices
42mNo Server Is Better Than No Server: Building Blocks for Serverless Architectures