A simple serverless data pipeline
summary
Small team builds a low‑ops, daily batch pipeline that moves megabytes per year from app storage into AWS S3, Glue, Athena; queries run in Athena (roughly $5 per TB scanned) and dashboards use Apache Superset (Preset hosted) for business users. Pipeline starts with CSV, tries ORC, settles on Parquet for compression and ecosystem tooling. No ETL runs — Glue only catalogues files into tables — and the setup proves cheap, simple and usable without a full‑time data engineer.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:12Architecture
- 2:45Context
- 3:52Team effort
- 4:19Recap
- 4:41AWS S3
- 5:28AWS Glue
- 7:08Athena
- 7:44Data Lake
- 8:49Data Sets
- 9:20Charts
- 10:15SQL Editor
- 13:24interoperability
- 15:09Conclusion
- 15:57References
- 17:02QA
related talks
42mBuild and automate a modern serverless data lake on
25mData Gone in 60 Seconds: A Serverless ETL Heist
38mUnleashing the serverless data pipelines
50mServerless SQL Pipelines
34mDesigning Data Intensive Applications In Serverless Architecture
47mBuilding Data Lake Platform Fully Serverless
38mServerless Data Streaming on AWS
43mData warehousing reinvented for today's needs