Platform Engineering for Modern Data Infrastructure
summary
Tracks the 15–20 year evolution from Hadoop (2006–2007) to cloud managed services and lakehouses, claims managed services cut operational work but create cost and vendor‑lock risks, and shows lakehouse tech like Redshift BigQuery Snowflake and Delta Lake Iceberg Hoodie as the path to unified storage plus governance. Proposes concrete practices: infra‑as‑code, self‑service APIs, chargeback plus autoscaling and S3 intelligent tiering, data‑quality rules, fine‑grained access, and architectures that scale from terabytes to petabytes.
AI-written summary · about
chapters (10)
- 0:00Introduction and Session Overview
- 0:49Evolution of Data Platform Architecture
- 3:13Core Platform Engineering Principles
- 5:36Self-Service Data Platforms
- 10:33Cost Optimization Strategies
- 13:25Automation and Configuration Management
- 14:48Monitoring, Observability, and Data Quality
- 16:34Security and Compliance
- 19:25Future-Proofing Data Platform Architecture
- 24:15Conclusion and Key Takeaways
related talks
11mBreaking Data Silos: Enterprise Data Management
41mModern Data Architectures – What They Are and How to Implement?
25mBuild open format data lakehouse
58mUnderstanding Big Data for Software Engineers
34mData Lakehouse: Who, Why and How
43mSoftware Quality As It Relates To Data
31mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
43mAnalytics capabilities architectural patterns: A journey for data-driven insights