Data Infrastructure in a Multi-Cloud Environment
summary
Datadog runs separate cloud-region instances to avoid cross-cloud egress and meet legal constraints, using Kafka as the ingestion backbone and ephemeral clusters managed by Mortar cluster lifecycle with spot instances. Batch ETL runs in Scala on Spark on Kubernetes, streaming and low-latency processing use Flink for low-latency, orchestration still uses Luigi while a new event-driven scheduler is piloted. Snowflake powers multi-region analytics, DBT replaces many bespoke pipelines, and an in-house Census metadata/lineage store tracks PII and data quality.
AI-written summary · about
chapters (9)
- 0:00Multi-cloud data intro
- 4:59Why Datadog chose multi-cloud deployment
- 7:50Strategy and environment separation
- 11:24Data engineering use cases
- 18:05Cluster lifecycle management
- 25:31Data governance and metadata
- 31:53Snowflake analytics in multi-cloud
- 34:25Multi-cloud lessons learned
- 51:20Q&A
related talks
34mTales of Data Architecture Evolution
49mData Platform Architecture, Evolution, and Philosophy
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
27mThe journey of Data Mesh and Compliance in a Multi-Regional Data Lake
28mFlexible Cloud, Whatever The Workload
17mDemocratising data analytics
34mCloudy With a Chance of Kubernetes: Going From One To Three Cloud Providers
01h 03mDiscovering the value of a cloud data platform