Stream vs. Batch: Leveraging M3 and Thanos for Real-Time Aggregation
Talkat KubeCon + CloudNativeCon North America 2021 · on Thursday, 2021-10-14· at 11:00· Room 403 AB + Online·
summary
Compares two concrete implementations: M3 uses streaming roll‑up rules on the ingest path to reconstitute aggregated metrics as new counters/gauges and serve them immediately, reducing an example dashboard from 51,000 → 230 series and query time from >20s → <1s, but adds operational complexity and loses arbitrary PromQL. Thanos uses Prometheus sidecar, ruler and compactor to run recording rules and write aggregates to object storage (2h blocks), keeping full PromQL support but incurring re-query/read/write overhead and slower, more resource‑heavy batches; notes a viable hybrid with M3 coordinator + Thanos.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:08High Cardinality Metrics Example
- 2:02Querying without aggregation
- 5:18Prometheus Recording Rules
- 6:18Open source metrics solutions
- 6:55What is M3?
- 8:20Streaming aggregation with M3
- 10:07Pros and cons of streaming aggregation
- 11:31What is Thanos?
- 13:29Batch aggregation with Thanos
- 15:13Pros and cons of batch aggregation
- 17:05Recap - How to choose?
related talks
29mStreaming Recording Rules for Prometheus, Thanos, and Cortex Using...
11mAggregating, Alerting and Graphing on Millions of Prometheus T...
31mHandling Billions of Metrics with Prometheus and Thanos
31mWhen Prometheus Can’t Take the Load Anymore
36mRunning the Observability As a Service For Your Teams With Thanos
26mPrometheus That Scales
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments