Modular & Reusable: A Recipe to Scaling Data Across a Portfolio
summary
A small central team of three data engineers builds a BigQuery-based ELT platform using Airflow (Astronomer) and dbt, onboarding eight e‑commerce companies from more than 60 sources with a single common data model packaged as a reusable dbt module and imported per project in one repository, using a hub-and-spoke Looker semantic layer and selective activation. Daily batch pipelines, DBTML for BigQuery ML and strict contracts enable rapid onboarding and deployable AI/ML features while keeping costs and headcount low.
AI-written summary · about
chapters (11)
- 0:00Modular data platform intro
- 2:13Agenda and background
- 5:41Solution architecture
- 7:27Scaling challenges
- 12:24Data transformation with dbt
- 19:25Common data model
- 24:03AI in data workflows
- 27:14Semantic layer with Looker
- 31:29Orchestration design
- 34:59Unified data model pros/cons
- 37:48Q&A
related talks
49mdbt: A Software Engineering Approach To The Analytics Workflow
39mBuilding a Modern Data Warehouse Without Breaking the Bank
23m‘Samen Slimmer met Data’ – A company wide Data Transformation
55mThe ELT pattern in action with DBT and SQLMesh
37mBuilding a Data Platform from scratch
47mWe need a Data Platform!
19mData science for real problems
31mBuilding 3D Trusted Data Pipelines With Dagster, Dbt, and Duckdb