Snowcannon: Synchronous Data Replication in FDB
summary
Snowcannon is a synchronous replication layer built inside FoundationDB: the producer cluster writes transactions to all TLog replicas and a majority of Snowcannon logs before commit, then Snowcannon asynchronously pushes batches to consumer clusters. Snowcannon implements append-only on-disk logs that often outperform TLogs, buffers terabytes so secondaries can be down for hours, enables a hot standby five seconds behind the producer for near-instant flush and zero data loss, at the cost of slightly higher commit latency.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:22OUTLINE
- 0:55SNOWFLAKE REQUIREMENTS
- 3:36COMPARISON OF REPLICATION SOLUTIONS
- 5:59HOW TO BUILD A DISTRIBUTED SYSTEM
- 7:19WE ADD A NEW MACHINE CLASS
- 7:33WE ADD SUPPORT TO RECRUIT THE NEW ROLE
- 7:56ADD SUPPORT TO THE WORKER
- 8:45CONCLUSIONS
- 9:21HIGH LEVEL VIEW OF ARCHITECTURE
- 9:59SNOWFLAKE DATA REPLICATION
- 14:30SNOWCANNON SWITCH
- 16:11CHALLENGES WITH PRODUCER RECOVERY
- 18:54SnowCannon, designed for Snowflake Metadata Requirements
related talks
11mNew Snowflake Capabilities and What They Mean for FoundationDB
22mSiphon : Modern Data Stack with SF-CH & Iceberg
34mFoundationDB, the Black Knight
40mHow DTCC achieves data resiliency with Snowflake and AWS (PRT272)
21mCaching Snowflake data with Aerospike for advanced analytics
11mConsistent Caching in FoundationDB
24mDatabase Native Support for CDC Streaming Pipeline
46mFoundationDB: The Best-Kept Secret of New Distributed Architectures!