Addressing the parallelism gap in replicated server systems
summary
Claim: backups can match or exceed primary parallelism by replaying logs with explicit dependency information rather than serializing. Method: build a directed conflict graph from log records (using conflict-driven scheduling), parallelize analysis by partitioning objects (partitioned dependency mapping), and expose state only at a low-water mark prefix plus incremental checkpoints. Concrete wins: avoids unbounded replication lag seen in single-threaded MySQL/PostgreSQL replication on 50–128+ core machines; implementation is nontrivial but practical and evaluated against prior deterministic-scheduling work.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:06Multi-core hardware is everywhere In the cloud
- 1:33Use advances in programming multi-cores Programming models
- 1:59Availability is money
- 2:15High availability via replication
- 3:43It's complicated: Non-determinism
- 4:38Replication requires determinism
- 5:10Setting expectations
- 5:34Assumptions
- 6:22What's in a log?
- 7:49Parallel log replay
- 8:43Replication has limited parallelism
- 10:31You don't have to take my word for it...
- 12:53Backup parallelism
- 15:19Serving reads on backups
- 19:49Implementation questions
- 20:16Parallel conflict analysis
- 22:04Executing reads
- 24:23Ideas with origins in deterministic databases
- 26:16Multi-cores and replication seem easy to get right
- 26:47Conclusions Server code must exploit multi-core hardware and remain highly available via replication
related talks
40mAll In With Determinism for Performance and Testing in Distributed Systems
40mwant don't You consistency no stinking
45m(...)Distributed Systems
27mBuilding Event-Driven Systems with PostgreSQL Logical Replication and Drasi
39mWhen "Worst" is Best (in Distributed Systems)
01h 00mThe Secret Life of Distributed Systems
40mUsing Multi-Master Data Replication for Refactoring via Parallel Run
34mTowards Effortless Transaction Management in Microservices