ParaSync: Exploiting Fine-Grained Parallelism for Efficient File Synchronization
summary
ParaSync breaks the CDC sync bottlenecks by (1) multi-threaded chunking that computes CRC32 per sub-chunk and merges checksums via checksum combination instead of re-reading data, (2) a streaming matcher that sends matching token batches as they form to overlap network and compute, and (3) an absolute-offset reconstructor that allows out‑of‑order, pipelined writes. Implemented on 16‑core NVMe instances and tested on Linux kernels, DB backups and VM snapshots over WAN (500 Mbps/50 ms) and LAN (10 Gbps/0.4 ms), it achieves up to 7.6× chunking speedup and up to 3.7× end‑to‑end with similar network use.
AI-written summary · about
chapters (10)
- 0:00ParaSync research intro
- 2:06Traditional sync basics
- 3:16Content-defined chunking problems
- 7:30ParaSync's parallelism approach
- 8:10Parallel chunking with checksums
- 9:18Streaming chunk matching
- 10:21Pipeline reconstructor offsets
- 11:48Performance evaluation
- 14:14ParaSync benefits summary
- 14:58Q&A
related talks
14mScaling Large Production Clusters with Partitioned Synchronization
29mAccelerating Filesystem Checking and Repair with pFSCK
14mSyncWise: Error-Aware Time Synchronization for Reconfigurable Data Center Networks
27mFrom Thundering Herd to Zero Outages: Building Reliable Inventory Sync
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
28mRock around the clock (synchronization
30mScaling HDFS with Consistent Reads from Standby Replicas
22mAsynchronous Computing at Scale: Overview and Learning