Jepsen 9: A Fsyncing Feeling
summary
Jepsen uses randomized clients, fault injection and history checkers to test real systems and finds concrete failures: Tendermint/MerkleEyes corrupts state under concurrent BTree GC and duplicate validators, GoLevelDB crashes on truncated writeahead logs, Hazelcast locks queues and atomic longs violate exclusivity and lose or duplicate items under partitions, Aerospike loses writes without durable flush and its tiny regime counter causes clock-skew recovery to resurrect older writes. Tests trigger and reproduce failures, vendors mostly patch bugs, and recommended fixes include proper WAL semantics and consensus (Raft/Paxos) for strong guarantees.
AI-written summary · about
chapters (9)
- 0:00Start
- 11:52Byzantine fault tolerant
- 17:18In-memory DATA
- 18:15Split- Brain Protection
- 20:03Atomic Reference "Guaranteed atomic compare and set across a cluster"
- 23:30Recommendations - Choose replication algo appropriate for data type
- 25:51Distributed, sharded {Document} store
- 34:37Recap
- 36:18Consider your failure modes
related talks
55mJepsen IV: Hope Springs Eternal
45mJepsen II: Linearizable Boogaloo
53mKeynote - Jepsen 7: Anna Concurrenina
41mCall me maybe Jepsen and flaky networks
43mPartitions for Everyone!
27mUnderstanding Distributed Systems with Deterministic Simulation
32mBuilding Reliable Infrastructure in Rust
41mWhat can go wrong with the distributed system - the Hazelcast exp...