Data engineers are from Mars, Platform engineers are from Venus how do I build
summary
Platform and data teams clash over one concrete thing: production data access. Example from a UK health‑tech startup in 2019 takes months to resolve; platform policy blocks raw access, data engineers claim mock data is useless. A rushed MLops build creates an ECS cluster and an S3 Star on Star IAM policy, causing a security incident, then gets fixed by involving core infra. Teams converge: analysts learn Terraform, pipelines run on Google Cloud and Vertex AI, and collaboration actually works.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:31DevOps history and gaps
- 5:00Data engineers' needs
- 7:52Production data access need
- 11:31Fulfilling engineer requirements
- 13:37Bridging communication gap
- 16:28Global collaboration vision
- 19:38Tunnel vision risks
- 22:25Trust in teams
- 24:11Conclusion and key takeaways
related talks
35mBuilding a data analytics platform on strong opinions held loosely
51mPanel Debate: Rise Of The Data Engineer
47mDataOps for AI-Ready Platforms
40mData Engineering in a Rapidly Changing World
25m"Data Engineering: The World Between Worlds"
45mDevOps Is Dead, Long Live Platform Engineering!
30mPlatform Oriented Data Architecture for Acceler...
38mThe Rise & Downfall of the Data Engineer REVISITED