Towards a Fault-Resilient Cloud Management Stack
summary
A custom fault-injection tool for OpenStack generates execution graphs, replays user requests, injects crashes, timeouts and network partitions, and checks both responses and internal DB invariants. Results show many practical failures: out‑of‑box single points of failure (DB, AMQP), missed timeouts, qpid client vs wrapper ping‑pong, sockets misinterpreted by select, and VMs stuck in BUILD forever; logging adds 1–2% overhead while replay is much slower. Recommendations include periodic cleanup checks and built‑in HA deployment.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:35What is a Cloud Management Stack
- 1:33Motivation
- 3:04Setting the Scene
- 3:44Failures are the norm
- 4:16Initial study
- 4:52Overview
- 5:08Openstack
- 5:59FaultResilient
- 6:54Fault Injection
- 7:32Execution Graph
- 11:02Fault Injection Results
- 11:27High Availability
- 12:51Timeout mechanisms
- 14:04Cross layer coordination
- 16:09Periodic checks
- 18:00Conclusion
related talks
25mChaos Engineering for Cloud native Apps
21mSponsored Session: Fault-Tolerant Training: How We Build Rel...
12mExponential Load Testing...
20mTaking Cloud to Extremes: Scaled-Down, Highly-Available, and Mission-Critical
29mLessons Learnt Operating Multiple Cloud Foundry on OpenStack Deployments
52mLet’s build a private cloud – how hard can it be?
39mResilience & Chaos Engineering in the Cloud
36mAdventures in Running a Private Cloud