Data - The Land DevOps Forgot
summary
Operational analytics handles petabytes per day with a 100,000-node DAG that runs on about 15,000 EC2 instances and nightly batches that now take three to four days, while the service side runs ~3,500 microservices; codebase includes 3.3M lines of Scala and 60 PRs/day. The fix is data mesh: decentralize ownership, add public/private data flags, route OpsGenie alerts to product owners, allocate compute costs back to domains, build a self-service platform, and accept major gaps in vendor interoperability and metadata standards.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:11Operational vs analytic
- 6:35The opposite of DevOps
- 8:36Even more challenges
- 9:36Hope on the horizon
- 11:07Is that actually a new paradigm?
- 11:27Data mesh
- 13:22Decentralize ownership
- 16:36Good fences make good neighbors
- 24:20Simple idea...hard to implement
- 34:31Missing parts
- 38:10How to get there?
- 45:03Summary
- 46:10Outro
related talks
28mDevOps in a Data Warehouse: Inside Out
47mDataOps for AI-Ready Platforms
33mAccidental DataOps: How we matured our organization with data, and how you can too!
35mBuilding a data analytics platform on strong opinions held loosely
47mAll the DataOps, all the paradigms
20mDevDataOps - Using data for exponential growth of your organization
51mFrom DevOps for data to DataOps
07mFrom DevOps to DataOps