0Flake - Reaching reliable non-flaky tests
summary
Finds three root causes of flakiness: unit floating point precision, non‑deterministic ML integration, and eventual‑consistency pipeline hiccups. Replaces brittle assertions with domain tolerances like +/-1 mile tolerance for distance tests, moves high‑value checks out of CI into a preproduction "canary on steroids" by deploying a fully scaled blue environment behind an affectless toggle and replaying real traffic for ~10–15 minutes, and builds a reconciler with damping that detects, limits and resends missing events; the pipeline reduces false failures and raises confidence in releases.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:16Flaky tests can fail or pass for the same configuration
- 0:39patch the test ... and test the patch
- 0:59Confidence is not binary, it's a spectrum
- 4:24Fraud Prevention Decision as a Service
- 7:58Unit Testing: Floats
- 9:02Unit Testing: Geo Distance Precision
- 10:33Expected results is a range, not a single value
- 11:41Non-Deterministic Results Flaky Integration Tests
- 12:00Features report exceptions but use fallback logic
- 13:26Integration Testing Stability vs Coverage
- 15:40Retries
- 16:07Docker sidecars
- 16:53"Monitor" exceptions raised during each test
- 20:12Why canary is not the right choice for us
- 24:22Fraud Analysts
- 24:57Developers can force ELB switch
- 27:50Data Pipeline Hiccups Flaky System tests
- 30:03Naive system tests (sleep 60)
- 30:56Continuous Data Reconciliation
- 34:10Reconciliation Caveats
related talks
22mTest Confidence: Types of Flake & How to Remediate
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
46mThe state of the Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
42mDon't Let Your Tests Flake Out
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
15mHow to (not) write a flaky test
47mTesting on Thin Ice: Chipping Away at Test Unpredictability