Building a Real Time Metrics Database
summary
Datadog processes trillions of metrics per day and is architecting for a quadrillion, combining Cassandra, Kafka, PostgreSQL with in‑house in‑memory stores to serve real‑time distribution metrics. Intake fans out into Kafka partitions (customer → metric → dynamic sharding), reads into RAM/Redis caches, local SSD nodes for days of data and S3 for long term, and uses heartbeat probes to avoid false alerts. Percentiles use a 4,000‑bucket relative‑error sketch (open‑sourced) instead of full sorting.
AI-written summary · about
chapters (11)
- 0:00Introduction to metrics infrastructure
- 1:27Trillions of metrics per day
- 3:11Performance mantras for scaling
- 6:25Architecture overview and caching
- 8:29Data storage choices and trade-offs
- 16:42Hybrid storage strategy
- 17:51Synchronization and heartbeat pattern
- 19:36Approximation for deeper insights
- 29:51Engineering triangle and sketches
- 31:58Commutativity and flexible architecture
- 34:03Summary and key takeaways
related talks
32mBuilding Real-time Systems with Open Source Technologies
11mBig Data Analytics: Immutability & Idempotence
48mHyper Speed! When Big Data Blooms
47mBuilding a Real-Time Analytics Database
29mAI Driven Observability based on Open Source
27mMetric Store
40mBuild Real-time Analytic Applications: The Easy Way
49mThinking in time series, the new solution to your pro...