Testing in A Distributed Systems World
summary
Argues that tracing’s context propagation becomes a testing primitive: attach an OpenTracing/OpenTelemetry baggage like tenancy=test to requests, route them (Envoy/nginx or service mesh) to separate test instances or use multi-tenant storage, mark metrics/logs as test telemetry, and run per-request failure injection and feature-flagged paths against production after deploy but before release. Shows Go API examples, notes current HTTP-header string-matching is required, and warns about auth, telemetry pollution and state poisoning.
AI-written summary · about
chapters (19)
- 0:00About me
- 1:06Cost of solving an issue
- 3:15Testing monoliths
- 3:47Testing microservices
- 6:18Risks of testing in production
- 7:34Test tenancy
- 9:46Context propagation
- 12:55Example: integration testing
- 13:26Instrumenting tests
- 13:52Managing state
- 15:31Managing telemetry data
- 16:18Managing other side effects
- 17:05Example: shadow e2e testing
- 18:07Example: fault injection testing
- 19:11Example: feature flagging
- 19:47Example: test accounts
- 20:39Consequences
- 22:35Key takeaways
- 26:12Types of tests in production
related talks
16mDistributed Tracing is Awesome, Testing is Not... Until Now
36mHow Testability Drives Observability
57mYour Distributed System is Talking to You: OpenTelemetry
07mWhy Tracing is a Lifesaver for E2E Tests of Distributed Systems?
39mHow Testability Drives Observability
31mRepurposing OpenTelemetry Traces as Test Data: Breaking the Cost Barrier in System…
32mDistributed Tracing in Action: Our Journey with OpenTelemetry
01h 02mOpenTelemetry Tracing — Real Challenges in Complex Systems