Instrumenting mayhem: functional chaos engineering
summary
Describes a working Elixir-based chaos platform called typhoon that runs paired load tests, injects faults by spawning anonymous functions on the Erlang VM or by system tools, and compares steady-state metrics. Experiments include kill worker pool to simulate DB outage (service stays up, latency drops), mocking auth calls to add delay (logins fail because of missing retries), and an iptables-induced network drop to Kafka (causes duplicate messages due to double retries). Tools: Elixir, Erlang VM, Kafka, Docker, iptables, load testers.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:00What is chaos engineering
- 6:55How to practice chaos engineering
- 14:35Database failure
- 15:25What we did
- 16:10Hypothesis
- 17:55Solution
- 18:55Results
- 22:15Automation
- 23:30Recap
- 23:55Typhoon
- 24:25Infrastructure
- 28:05Summary
- 28:30Thank you
- 28:45Question
- 35:55Failures
related talks
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
46mDesigning a System for Chaos
57mChaos Engineering Primer
33mMonkey Business For Serious Systems: Why chaos engineering isnt crazy
49mChaos Engineering: Withstanding Turbulent Conditions in Production
37mCreating chaos in containers
52mGetting started with Chaos Engineering
23mHow to tame chaos effectively?