Cassandra at Yammer #bbuzz
summary
Yammer migrates its inbox index from a Berkeley DB backend to Cassandra, using shadow deploys, double dispatch and integration tests to validate semantics. The inbox is read-heavy — half a billion reads a day with spikes that can fan out to 300,000 inboxes — so the first Cassandra schema (secondary indices, CRDTs) fails under migration load. Redesigned to partition by inbox/thread, use leveled compaction, probabilistic asynchronous trimming, shorter tombstone grace and frequent repairs, achieving P99 250ms but triggering a major outage three months later that prompts bulkhead timeouts and operational hardening.
AI-written summary · about
chapters (16)
- 0:00Introduction and Yammer overview
- 1:12Problem with legacy inbox service
- 4:26Why extract the inbox feature
- 7:33Methodology and migration approach
- 9:19Choosing Cassandra over Riak
- 11:26First Cassandra data model attempt
- 14:55Redesigning for Cassandra strengths
- 17:44Understanding Cassandra compaction
- 19:50Optimizing unread count queries
- 21:59Production outage and root cause
- 23:03Fixing compaction and trimming issues
- 25:31Implementing bulkhead pattern
- 27:16Failure isolation and impact
- 28:43What worked well in the rollout
- 30:09Lessons learned with new tech
- 31:32Q&A
related talks
29mCassandra - an introduction
35mThe What and Why of Cassandra Datamodelling at Scale
32mMigrating Workloads from Mainframes to Cassandra - Problems, Experience and Learnings
23mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
40mCassandra powered workflows to automate at scale
25mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
35mReplacing Cassandra: A Digital Transformation for the World’s Largest Digital Identity Network
40mLatency, Throughput & Fault Tolerance: Building a Modern Database