The Distributed Database Behind Twitter
summary
Tracks Twitter from MySQL to Cassandra to a custom NoSQL called Manhattan built 2012–2014 to solve Cassandra’s gossip scaling, missing counters and heavy ops cost. Manhattan runs tens of millions of requests/sec over petabytes on tens of thousands of nodes (largest >4,000), hosts >1,000 databases in ~20 clusters, and supports multiple backends including RocksDB backend, SSTable, B‑tree and a read‑optimized CDB. Architecture uses leaderless replication with last‑write‑wins, tunable consistency and an optional Kafka‑style replicated log for per‑partition linearizability; operations are automated by Genie cluster manager.
AI-written summary · about
chapters (20)
- 0:00Start
- 0:39MySQL
- 1:532008-2009
- 9:58Data Processing At Scale
- 11:49Design Principles
- 13:11Tech Stack
- 13:42Data Model
- 14:01Composite Structures
- 15:19Read Your Write Consistency
- 16:12Inconsistency Repair
- 16:27Storage Engines
- 17:35Strong Consistency
- 19:27Global Secondary Index
- 20:05Automated Imports and Exports
- 21:04Manhattan Scale
- 21:52Small Scale Operations
- 22:54Large Scale Operations
- 24:38Automated Operations
- 25:472020+
- 29:38Secondary Indexing
related talks
35mStorage, Then and Now
40mProviding Flexible Database Consistency Levels with Manhattan at Twitter
34mBig and Small Data at @Twitter
27mCloning Twitter: Rails + Cassandra = Scalable Sharing
30mThe Distributed SQL Database Behind Twitter
28mAn Architecture to Tweet Them All
01h 06mThe Legends of Twitter and Beyond: Real-World Architectures at Scale Panel
29mCassandra - an introduction