SkySync: Accelerating File Synchronization with Collaborative Delta Generation
summary
SkySync reuses storage-layer checksums to avoid recomputing chunk digests, implements CRC32 checksum combining to derive misaligned content-defined chunks, and replaces multi-stage lookups with a flat cuckoo-hash search. It is implemented by enhancing rsync/dsync protocols and extracting metadata via user-space tools, APIs, or small parsers. Evaluation on Alibaba Cloud shows up to 89.3% reduction in checksum computation, 1.1×–2× end-to-end speedup, and no increase in network traffic.
AI-written summary · about
chapters (11)
- 0:00SkySync and sky computing
- 1:03Traditional delta sync challenges
- 2:32Content-defined chunking limits
- 3:36Hardware acceleration and metadata reuse
- 5:04Collaborative metadata extraction
- 7:10Fixed-size chunking with metadata
- 8:15Fast checksum combining
- 10:25Trunk search optimization
- 11:06Heterogeneous storage protocol
- 12:12Experimental results
- 13:14Conclusion
related talks
29mThe Future of Real-time | Offline | Data
18mTeleport Data: The future of data
32mSupercharging IDE and Sync Performance
30mExpectations of an XaaS consumer
41mBuilding a simple-to-use Database Management tool
21mbeyond indexers trustless application data snapshots
33mWhat we learnt about DX from building
58mAnchored metadata