Chaos Engineering at Scale
summary
PayPal builds an internal chaos pipeline on Mangle (VMware), adds ticketing and ACL integration, then runs staged and production game days that inject packet loss to force fallback between service A → D → R. Experiments validate hypotheses like “P95 latency <200 ms,” compute a weighted resiliency score from observability queries, and reveal real bugs — e.g., a leaking connection that causes memory to climb over 5–10 minutes despite health checks. Faults run via an automated control plane, use kill switches and rollback, and scale by converting manual scripts into repeatable custom faults.
AI-written summary · about
chapters (11)
- 0:00Chaos engineering intro
- 4:42What is chaos engineering
- 7:57Chaos engineering principles
- 12:13Common faults in chaos
- 15:47Prioritizing chaos experiments
- 17:54Prerequisites for chaos
- 19:40What is a chaos game day
- 23:53Service resiliency case study
- 28:54Post-experiment analysis
- 32:21Chaos platform and tooling
- 36:41Key takeaways
related talks
38mChaos Engineering: A Step Towards Resilience
57mChaos Engineering Primer
46mDesigning a System for Chaos
33mImprove resilience with automated chaos engineering
49mChaos Engineering: Withstanding Turbulent Conditions in Production
48mChaos Engineering: The art of breaking stuff
37mBringing order to the chaos!
01h 11mChaos Engineering: Unleash the Monkeys