Scalog: Seamless Reconfiguration and Total Order in a Scalable Shared Log
summary
Scalog implements a shared log as per-shard local ordering plus an aggregator-based ordering layer that builds a global cut vector and uses a deterministic mapping to assign sequence numbers, so reconfiguration adds shards instantly without pauses. Implemented on 10Gbps servers with SSDs and two replicas per shard, it sustains roughly 52M 4KB records/s (ordering-layer limit, ~1.6 ms estimated latency) and scales to thousands of shards; compared to Corfu it avoids a ~30 ms reconfiguration hiccup and far outperforms Corfu’s ~600k writes/s saturation.
AI-written summary · about
chapters (10)
- 0:00Introduction to Skellig
- 0:39Shared log fundamentals
- 2:16Challenges in log design
- 3:35Centralized bottlenecks
- 5:20Skellig's hybrid approach
- 6:45System architecture
- 11:04Fault tolerance design
- 12:06Evaluation and results
- 14:13Conclusion and future work
- 15:44Q&A session
related talks
17mLow End-to-End Latency atop a Speculative Shared Log with Fix-Ante Ordering
21mLogDevice At Scale
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
01h 12mScaling distributed ledgers: Achieving arbitrary throughput and sub-second latency
39mScaling OpenSearch: Next-Generation Shard Allo...
45mOK Log: Distributed and Coördination-Free Logging
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
52mThe Computer Science behind a modern distributed data store