FinOps, how data engineers get their cloud cost under control
summary
A data warehouse of ~200 TB moves to Synapse and ends up in the top-cost list at €150,000/month, driven by a DW6000c compute tier and many expensive queries; one single SELECT TOP 25,000 forces massive scatter/gather and throttles concurrency. Pipelines built with Data Factory and Databricks show misuses: a naive ADF copy of 50 TB costs ADF 50TB copy €4,000, and long-running Databricks API pulls become cheap containers via move-to-container and orchestration changes. Continuous refactor, query profiling, and right-sizing compute are the concrete fixes.
AI-written summary · about
chapters (15)
- 0:00FinOps and cloud cost challenges
- 5:02FinOps vs traditional finance
- 7:50Making costs visible and finding waste
- 10:42Overcoming resistance to savings
- 13:11Chargeback vs showback strategies
- 16:23Data plane cost dangers
- 18:48Synapse performance pitfalls
- 22:05Refactoring Synapse compute costs
- 24:57Optimizing Databricks and API workflows
- 27:44Continuous refactoring and tech evolution
- 30:14Data movement mistakes in ADF
- 33:50Azure cost traps
- 35:33Predicting cloud costs
- 37:17Bridging data engineers and finance
- 40:28Q&A
related talks
39mBuilding a Modern Data Warehouse Without Breaking the Bank
50mFrom SSIS to Synapse to Fabric - A Data Architects Case Study
23mData Team as an Optimisation Problem
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
37mCloud warehouse monitoring ( Snowflake case study )
25mData Demons: Unleashing Chaos or Driving Performance?
01h 03mDiscovering the value of a cloud data platform
50mWhen and how to migrate to Azure Synapse?