Test trend analysis: towards robust, reliable and timely tests
summary
Proposal: instrument test results and store them as time‑series so flaky and slow behavior shows up as trends rather than one-off noise. The system posts JSON from test runs in a bulk upload to elasticsearch, visualizes via Cabana, and flags the top recurring error messages and tests using the 90th percentile timing. Result: a dashboard that identifies consistently flaky tests, common failure causes, and outlier slow tests, exportable as CSV and usable for email alerts.
AI-written summary · about
chapters (7)
- 0:00Start
- 11:59Non Determinism
- 16:55Elastic Search
- 20:18Top Slowest Tests
- 22:27Ui Tests as Change Indicators
- 31:09Storing Information over Time
- 31:57Summary
related talks
55mThe art of living with broken things
28mTest flakiness; methods for dealing with flaky tests
44mObservability meets a Flaky Test
46mThe state of the Art in Tackling Flaky Tests
17mContinuous Testing at Scale
28mThe State of Art in Tackling Flaky Tests
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
15mHow to (not) write a flaky test