Data Unleashed: A Developer's Perspective on Navigating the Architecture Maze
summary
Recommendation: start with your existing database as the baseline, ingest quick Excel/API feeds into the app, and only move to a warehouse when multiple teams reuse datasets; prefer a lakehouse over lake+warehouse to reduce duplication and complexity. Measure decisions with speed, cost, skills, scale, latency and AI integration potential, target 10x ROI, ship new datasets within months not years, use Databricks/Fabric/Redshift/Azure but monitor cloud costs closely because latency and surprises kill value.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:30Audience mix
- 3:19Data value speed trust
- 11:49Raw data insights
- 15:23Data architecture players
- 20:02Legacy databases challenges
- 23:17Database metrics baseline
- 25:24Avoid over-engineering
- 28:56Moving to data warehouses
- 31:45Warehouse patterns
- 33:33Database to warehouse trade-offs
- 35:21Data lakes vs lake houses
- 37:10Lake houses simplify
- 39:38Data mesh microservices
- 41:47Lake house mesh teams
- 43:53Guiding principles
- 45:19Start simple evolve
- 46:43Cloud cost management
- 48:07Latency kills value
- 49:12MVPs user testing
- 49:55Embrace monolithic
- 51:01Cloud cost optimization
- 51:47AI catalyst
- 53:17Q&A
related talks
31mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
43mAnalytics capabilities architectural patterns: A journey for data-driven insights
44mAgile Data Warehousing/Business Intelligence: Addressing the Hard Problems
58mTales of Data Architecture Evolution
25mPlatform Engineering for Modern Data Infrastructure
58mUnderstanding Big Data for Software Engineers
11mBreaking Data Silos: Enterprise Data Management
31mData Warehouses Meet Data Lakes