TestOps at scale
summary
Nightly runs execute end-to-end and unit tests across 30+ machine targets and many branches, using behave (Python) for Gherkin specs, Catch2 for C++ component tests, and Jenkins lockable resources to queue access to scarce hardware. Test results stream into an ELK analytics pipeline for signature-based failure grouping, dashboards and trend graphs; an internal React GUI orchestrates runs and is being open-sourced. The rig debugs locally, reduces false negatives, and provides rapid feedback to developers.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:48Survey
- 1:32Intro / Disclaimer
- 2:42Is this talk for you?
- 4:34Testing as part of the development effort
- 5:13The way we put "Test as Activity" into practice
- 9:08Same test automated with Behave
- 10:05Labor division
- 10:59Agile, actually
- 12:19Gherkins superpower
- 15:58Test review tips
- 21:30Jenkins test run sequence
- 22:58The same platform all the way through
- 24:12Change traceability
- 24:44Test at the right level
- 27:44More thoughts about Test Ops
- 28:58Dashboards and analytics
- 31:44Guiding questions for Search and Dashboards
- 34:06Search and analytics value proposition
- 35:02The mental shift
- 37:06Long process to increase the value of logging, search & analytics
- 39:28Test coverage
- 47:38Questions?
related talks
15mTransforming Testing: The Power of TestOps
30mTestOps: Transforming Chaos into Streamlined Story-Based Releases
31m#testedAF: Test All the Things (Not Just Your Units)
16mTesting All The Way To Production
43mPut your TestOps shoes on
46mTestOps: A Journey to Story-Based Releases
40mPer Aspera ad Excellent TestOps
36mPut Your TestOps Shoes on! Improving Quality by Process Automation