Do NOT click-ops your data infrastructure
summary
Argues that databases and streaming platforms must be provisioned with infrastructure-as-code rather than clicks, using Terraform to declare resources, variables and naming conventions for reproducibility and repeatability. Demo provisions a source Kafka cluster, a target cluster and an Apache Kafka MirrorMaker replication flow across regions/providers (three services created, demo time-lapsed 8x) and shows terraform plan/apply lifecycle. Covers version upgrade and config-change tradeoffs, warns about public module misconfiguration and recommends research, pilot and staged rollout.
AI-written summary · about
chapters (11)
- 0:00Welcome and introduction
- 2:33Why data infrastructure matters
- 4:40Infrastructure as code principles
- 7:08Challenges of infrastructure as code
- 9:56Handling unicorns in data infrastructure
- 12:23Introduction to Terraform
- 15:40Demo setup overview
- 16:44Terraform demo walkthrough
- 21:24How to get started
- 23:33Recap and references
- 24:58Q&A and closing
related talks
44mTaming the cloud sprawl
27mTerraforming Data Analytics Environments
48mMoving from spaghetti infrastructure to composable cloud environments
30mPlatform Oriented Data Architecture for Acceler...
01h 32mUsing Terraform and AWS to create environments on the fly
27mCase Study: Re-Thinking Our Infrastructure Tooling
39mTerraform Practices to Enable Infrastructure Scaling
01h 01mTaming the Modern Data Center