Natalie Godec DevopsDays Geneva 2020
summary
A production data platform runs Kafka -> BigQuery with transformations scheduled in Cloud Composer (managed Airflow); pipelines are defined as Python DAGs, run hourly, and a demo DAG wrote a new table successfully. Infrastructure is deployed with Terraform, CI/CD applies PRs, and cross‑cloud VPNs move data into GCP. Access is segregated with project-level IAM plus VPC Service Controls, prod data is immutable and researchers use separate workspaces. Team footprint: ~2 infra engineers, 5–6 data engineers, and ~40–60 analysts. BigQuery separated storage/compute
AI-written summary · about
chapters (12)
- 0:00Introduction and mission
- 4:37Why data matters now
- 6:21Challenges with legacy systems
- 9:10Introducing BigQuery
- 11:15Babylon’s data pipeline
- 13:02Cloud Composer and Airflow
- 19:54Terraform demo walkthrough
- 24:50Building DAGs in Airflow
- 31:17Securing sensitive data
- 36:14Breaking data silos
- 38:23Future of data platforms
- 39:11Q&A session
related talks
26mETL pipeline with GCP Dataflow
17mDemocratising data analytics
30mData Warehouse in Google Cloud
15mData strategy: Getting ready for Cloud migrations
01h 01mData Infrastructure in a Multi-Cloud Environment
31mA Journey of Migrating Data Platform from Data Center to Cloud
19mKickstarting a greenfield data project
40mFinOps Examples using Google Cloud BigQuery by Aliz.ai