Kickstarting a greenfield data project
summary
Greenfield platform centralizes fragmented systems into an immutable Azure Blob Storage data lake (parquet, hive-style time partitions, cold-tier after one year) while selectively promoting datasets into a Snowflake warehouse for self-service BI (Tableau/Power BI). Orchestration runs via Apache Airflow in Docker with CI/CD, macros and Slack alerts. Data volume expects to grow more than threefold in 12 months, current team is two people, and planned additions include Amundsen for discovery and Great Expectations for validation.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:49ABOUT ME
- 2:33Where we started...
- 8:04Foundational Work
- 9:11Data Fragmentation
- 11:32Curtailed Data Scalability
- 13:17Legacy Processes
- 14:45Enterprise Data Platform
- 16:01Learnings & Pitfalls
- 17:25Coming up.
related talks
41mOn Data: Sky above, sand below, peace within
17mDemocratising data analytics
35mBuilding a data analytics platform on strong opinions held loosely
30mData Engineering building your BI infrastructure from scratch
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
34mTales of Data Architecture Evolution
45mHow Data Hub flies above the clouds!
24mREDLake Azure - Enterprise Data Lake and Business Satellites