Building a Data Platform from scratch
summary
Starts from a Postgres source on Google Cloud and targets terabytes of events; chooses GCP as the cloud, picks BigQuery as the warehouse, uses Spark for heavy transforms and Airflow for orchestration, and self-hosts change‑data‑capture after Google Data Stream proves cost‑prohibitive so a pdb self‑hosted on Kubernetes handles CDC. Uses metabase for metabase embedded analytics via iframe and Airbyte for imports/exports. The platform delivers embedded analytics and export pipelines in production and lowers ongoing costs compared with managed alternatives.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:36Startups need data platform early
- 7:49Key components
- 13:31Self-hosting vs managed
- 15:58Building at a startup
- 23:34Adding value with platform
- 27:45Scaling best practices
- 29:33Q&A on AI costs
- 33:09Justifying costs to business
- 34:58BigQuery vs Spark vs Airflow vs Databricks
- 37:05Closing
related talks
19mData science for real problems
17mDemocratising data analytics
30mData Engineering building your BI infrastructure from scratch
36mMigrating Our Data Warehouse to the Cloud
35mBuilding a data analytics platform on strong opinions held loosely
40mDeveloping on Google Cloud with Python and DataFrames
47mData Warehousing Migrations: Lessons from Home Depot
28mBuilding Self-Service Data Platforms