Distributed system fault injection testing with Docker
summary
Built a Docker-based fault-injection test framework that runs locally or from AWS ECR, wires up multi-node Elasticsearch and Zookeeper with a lightweight elastic proxy and sidecar for dynamic routing, and automates tests with BATS. It uses Pumba to drive netem/tc faults (examples: Pumba, 8% packet loss, 800 ms delay). Results show packet loss hurts availability far more than latency, proxy adds ~3–4 ms, and the setup catches partition/brain-split cases before production.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:29Agenda
- 1:02Microservices
- 1:16Cap CRM
- 2:21Cloud
- 2:48Network
- 5:19Base
- 8:16Indirection
- 8:28Elastic Proxy
- 10:03Testing Microservices
- 11:26Nondeterministic world
- 12:27Space complexity
- 13:44Netflix
- 15:39Kirk Campos
- 16:32Demo
- 19:06Bash
- 19:44Bash like script
- 20:45Pumbaa
- 22:01Pumbaa up
- 22:55Identifying failing points
- 23:26Enabling package loss
- 23:57Cap theorem
- 26:15Single test scenario
- 28:36Summary
related talks
19mCAFault: Enhance Fault Injection Technique in Practical Distributed Systems via...
40mResilience Testing - Failure is an Option
34mChaos in the cloud
37mFrom mocking to rocking your tests with testcontainers
38mCreating chaos in containers
32mUsing Docker in test automation
28mPostmortem: TDD with TestContainers
46mPerformance Testing Framework