PromQL Compatibility Testing - Verifying Vendor Claims for End User Benefit
summary
A PromQL compliance tester runs 500+ real queries against a Prometheus reference and a vendor API, compares results (with allowances for floats and sorting) and reports pass rates and diffs. December 2020 runs show many systems at 100% passes (Chronosphere, Cortex, Thanos, M3, Promscale), New Relic scores ~31% and VictoriaMetrics ~60% due to divergent behaviors (timestamp alignment, removed staleness markers, different range-vector sample counts and NaN handling). Test reports include per-query diffs and configurable query tweaks for cross-cutting parser bugs.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:53Compatibility List
- 1:23Examples
- 2:14Compatibility Expectations
- 5:14PromQL
- 5:59Projects
- 6:22The Problem
- 7:10PromQL Compliance Tester
- 8:49Test Summary
- 9:28Project Donation
- 10:05Initial Test Runs
- 10:43Vendor List
- 11:38Interpreting Test Scores
- 12:43Detailed Results
- 14:19Chronosphere Overview
- 15:47M3 Overview
- 15:57Fire Overview
- 16:24New Relic Overview
- 18:04Promscale Overview
- 18:23Open Source PromQL
- 20:05Conclusion
- 21:19Future Work
related talks
10m⚡ Lightning Talk: Vote on Your Favorite--Standardize This!--10 PromQL Queries That Measure CPU & Memory Usage
30mDistributing PromQL for Fast and Efficient Kubernetes Fleet Monitor...
11h 43mPercona Community Live Conference - Day 1 - Open Source Databases Conference
34mSpecifics of Data Analysis in Time Series Databases
33mMetrics as a First-Class Citizen in the E2E Testing Landscape
31mTests Are What You Want, What You Really Really Want
41mProxySQL 2021 Dev Submit
28mChaos Validation Made Easy