Manifesto for a DevOps-like Culture Shift in Data & Analytics
summary
Scout24 adopts a seven‑point manifesto that redistributes responsibilities: a small central team (eight people) builds a cloud data platform on S3 and Kinesis Firehose while product teams publish events via Kinesis Firehose and document schemas in a data catalog. Data producers own data quality; data consumers design metrics and build materialized views from the lake. Platform features—automatic table publishing, backups, access control, parquet conversion—nudge teams to participate. The aim is a cultural shift called DataDevOps where the central data team owns platform but not the data.
AI-written summary · about
chapters (5)
- 0:00Start
- 15:10Central Data Team's Responsibility We, Data & Analytics, are responsible for providing a solid Data Platform as wel as clear guidelines and training how to participate in the Data Landscape
- 16:44Data Autonomy, Not Anarchy Data autonomy puts data producers & data consumers in control of their data & of their metrics and thereby allows us to be data-driven at scale, but this comes with responsibility
- 23:08Consumer's Responsibility Data consumers are responsible for the definition & visualization of metrics and for driving the implementation and maintenance of these metrics.
- 39:48Design 'nudges' into the Platform Make Data Lake easier than something else automatic table publishing, partition detection backup and disaster recovery access control for restricted data optimize file formats de parquet for efficiency
related talks
30mDevOps For Data Teams
28mDevOps in a Data Warehouse: Inside Out
26mDevOps in a Data Warehouse: Inside Out
30mA Devops Approach to Data: A Tale Through the Trenches in Search of a Cloud Data Revolution
35mBuilding a data analytics platform on strong opinions held loosely
26mApplying DevOps practices in Data and ML Engineering
33mAccidental DataOps: How we matured our organization with data, and how you can too!
33mData: The Land DevOps Forgot