How We Saved 5x Migrating from Aurora to FoundationDB
Talkat FoundationDB Summit 2019 · on Monday, 2019-11-18· at 16:30· Grand Ballroom 5, Marriott Marquis San Diego Marina·
summary
Segment migrates a Persona identity-resolution service from Aurora to FoundationDB and reports a 5x cost reduction while keeping serializable isolation and high read throughput. The data model uses direct key-value subspaces with mappings and merges (surrogate-object keys plus index ranges). Production runs on a heterogeneous three-tier cluster provisioned by Terraform and ECS; backups from S3 restore slowly for multi-TB datasets, chaos tests show graceful recovery, and remaining concerns include data distribution unpredictability and triple-replication across three AZs.
AI-written summary · about
chapters (28)
- 0:00Intro
- 1:43What is CDI?
- 2:33Segment Timeline
- 3:18Identity Resolution Use Cases
- 3:29Problem
- 4:08Identity Graph
- 4:59General Data Model
- 5:41"Object" KV Kind
- 6:47Mapping Object
- 7:19Segment ID Index
- 7:48Merge Object
- 8:19Workload Profile Needs
- 9:18Personas Data Flow
- 10:47Aurora works, but...
- 11:17Some Baseline Aurora Costs
- 11:38Example Realistic Workload
- 12:58Tuning Cookbook
- 13:26Cluster Evolution
- 14:07Infrastructure Overview
- 15:18Terracode-foundationdb / cluster
- 15:55AMI Launch Config / User Data
- 16:30data.mount
- 16:44configure-fdb
- 17:28fdb-discovery
- 18:39fdb-backup
- 19:54Gameday & Chaos
- 22:22Exactly Once Delivery
- 23:23Dedupe Pros / Cons
related talks
22mSolving Everyday Data Problems with FoundationDB
37mTechnical Overview of FoundationDB
31mModernizing a megascale database
34mFoundationDB, the Black Knight
22mHow FoundationDB powers SnowflakeDB's metadata!
46mFoundationDB: The Best-Kept Secret of New Distributed Architectures!
26mRunning FDB at scale at Wavefront
06mLightning Talk: Performance Testing of FoundationDB using TPC-C