Migrating Datastores
summary
Buildkite migrates a single-threaded Redis cache to clustered Redis, splitting hot keys across three shards after hitting an 85% CPU cliff on a single instance. Engineering classifies keys into performance cache, capacity cache, and source-of-truth, then warms the new cache with a 1% traffic warmup rollout, adds percentage feature-rollouts and Redis-backed feature flags, and monitors shard CPU. Result: old node CPU peaks fall from the danger zone to ~60%, three shards share load, and customer-facing failures stop.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:25Buildkite Redis architecture
- 3:12Redis scaling limits
- 6:01Clustered Redis solution
- 8:29Data grouping for sharding
- 10:17Data types assessment
- 12:48Cache importance classification
- 14:54Migration strategy for caches
- 16:17Percentage-based rollouts
- 17:44Project side effects
- 19:07Migration results
- 21:37Key takeaways
- 23:22Project success
related talks
36mHot Swap Your Datastore: A Practical Approach and Lessons Learned
34mScaling Redis writes with cluster mode
59mUsing Redis beyond caching
31mUser Attribution at Scale: All-Flash, ½ Trillion Objects
48mRebuilding a RT Datastore: the design, deployment & performance story
24mMigrations under Production Load: How to Switch...
14mMigrating 15k Redis servers to Valkey
15mFrom Chaos to Clarity: Deciphering Cache Inconsistencies in a Distributed...