Better Testing Through Statistics
summary
OpenStack CI runs about 12,500 jobs a day, spinning up 5–25 clouds per patch and generating gigabytes of logs per run; the team stores logs in Graphite and ELK, loads structured test outcomes into subunit-to-sql, and visualizes runs with OpenStack Health and StackViz. An elastic-recheck system fingerprints failures and tracks ~150–200 recurring race/bug patterns; using these pipelines surfaces a performance regression (big jump in volume latency) that is identified and fixed within a week.
AI-written summary · about
chapters (14)
- 0:00Start
- 3:19Number of Tempest Tests per Day in the Gate Queue
- 3:50Log Server
- 5:54General Approach
- 6:45Graphite
- 7:47Grafana
- 9:05StackViz: Timeline
- 9:31Using OpenStack Health
- 10:47OpenStack-Health Architecture
- 13:07Elastic Recheck
- 15:52Data Driven Decision Making
- 16:56Finding trends amongst the noise
- 19:30Future work
- 20:46Where to get more information
related talks
32mTools for Better Testing for Open Source Projects
33mOpenStack Compute's automated testing
56mTestOps at scale
42mTesting Salesforce at Cloud Scale
34m1.5 Billion Tests Per Day
24mWill it Run on *My* OpenStack? – Building an Executable Validation Testsuite for CPIs
56mTesting at scale
32mBuilding Continuous Integration and Feedback