Data Team as an Optimisation Problem
summary
Company frames building a data practice as an optimization problem with weighted metrics and immutable vs mutable requirements, then applies it to concrete choices. Storage picks Snowflake for cloud‑agnostic usage billing; ETL uses Glue + PySpark with dbt for transformations and external Snowflake stages; visualization uses Metabase to keep costs low. Vendor tools like Fivetran/Stitch and QuickSight get rejected for cost or data‑sovereignty, Tableau remains a desired future. A one‑person team proves the pipeline works within budget and constraints.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:32Overall Architecture
- 4:56Data "Team"
- 5:44Whoami
- 6:58Data Landscape
- 7:55The Game
- 9:04Requirements
- 10:11Metrics
- 10:49Formal Definition
- 11:17Applying Theory
- 12:03Storage
- 13:38Amazon Redshift
- 15:53Visualisation
- 16:11Amazon QuickSight
- 16:44Tableau
- 17:39Metabase
- 18:26Extract, Transform, Load (ETL)
- 18:44Fivetran / Stitch
- 19:45Matillion
- 20:07AWS Glue
- 20:50Appendix #1: dbt Transformations
- 21:23Appendix #2: Snowflake External Stages
- 21:53Key Points
related talks
39mBREAKING YOUR DATA TEAM OUT OF THE SERVICE TRAP
31mFuture of data engineering: what's in store?
29mBuilding successful data teams in the realities of 2023
25mPerfecting Our Modern Data Stack Recipe: What We Learned On The Journey At Huel
50mMasterclass: Building a data stack in 2021
39mBuilding a Modern Data Warehouse Without Breaking the Bank
27mData team strategies: unleashing your team
27mHow To Develop & Operate An Agile Cloud Datawarehouse In A Rapidly Changing Multi-ERP Environment