Building a Canary Testing Framework
summary
A working canary framework that runs continuously in production claims to catch real regressions not visible to end‑to‑end suites: it uses a Go daemon to launch concurrent goroutines, embeds an embedded JavaScript VM (otto) for per‑test scripts, enforces per‑test timeouts and frequencies, and stores results in BoltDB storage while exporting Prometheus metrics and a GraphQL API. It runs per‑product canaries across regions, caught a surprising firewall race condition, and successfully logs structured test output.
AI-written summary · about
chapters (17)
- 0:00Building a Canary Testing Framework
- 0:25What is a canary?
- 1:07Canary tests should mimic user behavior and test a specific feature.
- 3:52Allow for more specificity and granularity for the execution environment
- 5:31Visualization of Test Output / Execution Status
- 5:49Recording and alerting on metrics via Prometheus
- 6:36Canary Framework Architecture Components
- 7:05A database for storing test execution results
- 7:18Why Go for the actual framework?
- 7:54Decent support for embedding languages
- 10:01Allows us to write and run test scripts using JavaScript
- 10:50The Canary Daemon
- 11:01Executes the config file and registers the tests to run
- 11:42Loading Test Configurations
- 18:21Standard Library Functions
- 26:12Saving Test Execution Results to a Database
- 36:11Questions?
related talks
21mI Don’t Always Test My Code, but When I Do It’s in Production
33mA Tale of Two Worlds: Canary-Testing for Both ML Models and Microservices
23mTesting in Production
10mLightning Talk: Open Source Canary Deployments with Application Metric Analysis
04mCanary Deployment With Prometheus
14mCanary Deployments for Infrastructure Upgrades
30mSRE's Swiss Army Knive - Canary Monitoring
22mTesting in Production