Inside Onyx
summary
Onyx 0.9.4 implements masterless coordination by storing a durable append‑only coordination log in ZooKeeper using sequential ZNodes; each peer keeps an in‑memory replica as value (immutable Clojure maps) and applies pure, idempotent functions from log entries to reach identical cluster state. Peers detect failures via a ring (each watches its neighbor), join with a three‑phase prepare/notify/accept protocol, GC by snapshotting an origin replica, and works well for hundreds of nodes though ZooKeeper contention and replica size limit larger-scale clusters.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:24What is Onyx
- 1:29Batch workloads
- 2:07Streaming workloads
- 2:44The payoff
- 3:13Why Onyx
- 4:50Coordination
- 5:51Master Workers
- 8:21Masterless System
- 14:22Log
- 17:35Replicas
- 18:05Logs
- 20:16Replica Console Viewer
- 23:31Failure Detection
- 28:58Resource Allocation
- 29:55Problems
- 30:22Replica Size
- 31:03Join Direct
- 31:57Log Activity
- 32:27Communication Interzone
- 34:19Gossip Protocol
- 34:50Questions
related talks
37mOnyx: Distributed Computing for Clojure
23mReaching Zen in ElasticSearch's Coordination
24mOxia - A Horizontally Scalable Alternative to Apache Zookeeper
33mStrong and Efficient Consistency with Consistency-Aware Durability
38mTowards consensus on distributed consensus #bbuzz
15mUnconference Session: Dedicated Coordination Nodes - Why? When? How?
45mIs there anybody out there?
25mCloud native data replication strategies