When the CFO Comes Knocking: Our Journey to a Cost Efficient Data Platform
summary
Claims a >50% cloud cost cut in under two years and shows how: add tags across AWS resources, pipeline billing into an Iceberg table, and attribute cost to teams and 8,000 pipelines by compute units; build a compaction daemon and a TTL/auto-delete process plus storage tiering via lifecycle rules (S3 Glacier) to archive cold partitions; tune compression (zstd), use Graviton instances, autoscaling and predictive scaling for Trino/Spark; result is a drop from eight‑digit to seven‑digit annual cloud spend and sustained cost visibility.
AI-written summary · about
chapters (20)
- 0:00Cost efficiency journey
- 1:29Why cost efficiency in 2022
- 2:52Visibility and accountability problems
- 4:14Four pillars for cost reduction
- 5:20Cost visibility via cloud tagging
- 9:12User attribution for team spending
- 13:27Cost drivers in data infra
- 14:31Cost-aware team practices
- 16:38Infrastructure guardrails for savings
- 17:42Storage optimization for data lakes
- 23:04Kafka compression strategies
- 24:47Data lake compaction
- 26:13Retention policies and tiering
- 27:41S3 lifecycle management
- 33:37Compute optimization for workloads
- 34:57Auto-scaling techniques
- 36:42Cost-effective compute types
- 38:28Results of cost reduction
- 39:32Ongoing cost efficiency culture
- 40:13Q&A
related talks
20mAtomics Habits in cost optimizations in the cloud
36mFinOps & Observability: technical implementation
23mFinOps and Cloud Cost Optimization
34mOptimizing Cloud Efficiency with Prompt Engineering
47mCost optimization
43mMigration to the Cloud, DevOps Perspective
19mFinOps for business results
38mBuilding Data Pipelines on AWS