Terraforming Data Analytics Environments
summary
Shows code that provisions a full analytics stack end to end: Terraform (Terraform Cloud backend) plus the Databricks resource provider and Azure RM to create a Databricks workspace, cluster, ADLS storage and a Kafka VM with a Solace connector, with post-provisioning via Ansible. CI runs in GitHub Actions and development uses GitHub Codespaces. Demo builds a working pipeline, deploys a notebook attached to the cluster, streams SWIM messages from Kafka, and finishes in about 15–25 minutes.
AI-written summary · about
chapters (14)
- 0:00Introduction and agenda
- 1:47Use case and architecture overview
- 3:32Challenges with manual setup
- 6:44Project structure and conventions
- 8:09Terraform deployment workflow
- 10:36Variables and tagging strategy
- 12:00Team collaboration with GitHub
- 14:24Databricks cluster provisioning
- 16:09GitHub Actions automation
- 17:35GitHub Codespaces for development
- 18:59Notebook deployment and testing
- 21:53Key takeaways and benefits
- 24:00Code highlights and integrations
- 25:22Closing and Q&A
related talks
26mBuilding the Ultimate Databricks Environment
19mBuilding out Tailwind Traders Data Mesh
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
26mDo NOT click-ops your data infrastructure
35mBuilding a data analytics platform on strong opinions held loosely
20mReady. Set. Automate! Have your Azure Databricks Lakehouse Up and Running in 20 Minutes!
44mFlying High with Data Engineering in Microsoft Fabric