Building a Modern Data Warehouse Without Breaking the Bank
summary
Company builds a usable data warehouse on $500/month, replacing Sunday manual extracts and spreadsheet reports with an Athena + S3 catalog, Airflow orchestration, Apache Superset visualizations and DBT models. A Slack request bot and weekly upskilling sessions create an analytics‑engineer culture; small tools—DBT coverage, DBT‑Superset sync, SQLFluff—enforce tests, docs and style. Later add a thin Data API, DuckDB and Iceberg for production queries. The setup runs under budget, survives acquisition and powers internal and external use.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:51Accidental warehouse necessity
- 6:52Data org on $500 budget
- 9:01Choosing warehousing tools
- 13:14Extracting unconventional data
- 14:39Selecting visualization layer
- 15:42Modern warehouse under $500?
- 16:24Slack bot for DMs
- 18:08Custom transformations with dbt
- 19:52Engineering practices for analytics
- 23:07Upskilling team gaps
- 25:16Analytics engineer role
- 28:26Future at scale prep
- 30:12Cisco acquisition scrappy
- 31:59Athena limits with DuckDB Iceberg
- 33:23Automating dashboards IaC
- 34:28Key takeaways
related talks
41mModular & Reusable: A Recipe to Scaling Data Across a Portfolio
58mTales of Data Architecture Evolution
23mData Team as an Optimisation Problem
32mHow to Build a Data Warehouse?
23mDelivering Modern Data Stacks With dbt to Operationalise Analytics
19mRobust CI Workflow with dbt for Data Warehouse Optimization
49mdbt: A Software Engineering Approach To The Analytics Workflow
34mTales of Data Architecture Evolution