What can go wrong in a distributed system - experience from...
summary
Hazelcast uses raw Java, a homegrown RPC and serialization stack, and a Raft-based consensus layer but an internal job snapshot uses an best-effort AP map so a forceful node shutdown causes corrupted or lost job state; replication needs linearizable guarantees or majority+linearizable ops. Discovery and message reordering let stale join requests evict live members, and a Raft slow follower plus an unbounded outbound queue produces OOMs under load. Engineering fixes include bounded buffers, retry limits and a flow-control fix that prevents the leader OOMs and restores stability.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 1:16What is Hazelcast
- 7:45Problems in replication
- 18:54Consistency guarantees
- 19:16Network implementation issues
- 26:50Out of memory exception
- 30:12Raft protocol problems
- 42:25Key takeaways and advice
- 45:17Final summary and Q&A
related talks
41mWhat can go wrong with the distributed system - the Hazelcast exp...
45mjava.util.concurrent for Distributed Coordination
47mDistributed Systems: When Democracy and Computer Science Come Together
47mjava.util.concurrent for Distributed Coordination
50mDo Distributed Systems they said. It'll be fun
30mDistributed Computing with Hazelcast
45mDo Distributed Systems they said. It'll be fun they said.
48mHow to build distributed concurrency primitives