Lightning Talk: Correlating Time Series at Scale
summary
Implements a published algorithm that previously scaled to 5,000 series and runs it against a Kubernetes testbed that produces 40–80,000 time series, completing correlation on a 2‑CPU, 4‑GB VM. Writes results to VictoriaMetrics or to “pocket” and hits ingestion and query limits around 15 million metric writes; finds 6–12 million correlated pairs most of which are trivial (constants and same‑slope counters). Identifies the real bottleneck as result writing/latency and explores parallel MapReduce partitioning and schema changes to reduce boring matches and surface the interesting correlations.
AI-written summary · about
related talks
36mVictoriaMetrics: scaling to 100 million metrics per second (EN)
13mLightning Talk: A Drop-in System To Accelerate Metrics Observability by 100x Us...
56mUnwiring High Cardinality
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
28mTick Tock: What the heck is time-series data?
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...
33mStory of Correlation: Integrating Thanos Metrics with Observabi...