Wikimedia Content API: A Cassandra Use-case
summary
Wikimedia serves ~18 billion monthly page views and builds a public content API that stores full-history rendered HTML for revisions using parsoid + restbase cache backed by Apache Cassandra 2.2.6. The production footprint is a 54-node cluster spanning two data centers on beefy hosts (16 cores, 128 GB) with 18 physical machines running multiple Cassandra processes, vnodes=256. Surprising operational notes: deflate with 512k chunking yields much better compression, compaction strategies (DTCS vs size tiered) and overlapping-table tombstones cause space pain, and G1 GC needs region-size tuning.
AI-written summary · about
chapters (18)
- 0:00Start
- 2:24LAMP
- 4:07WYSIWYG
- 4:39Conversion
- 5:02Character-based diffs
- 5:22Metadata
- 6:40Parsoid
- 7:54RESTBase
- 8:52Environment
- 10:45key-revision-value
- 12:31Data model
- 14:09Brotli compression
- 16:41Compaction strategies
- 21:05The Overlapping Tables Problem
- 23:42Humongous objects
- 25:12Motivation
- 26:24What we do
- 27:07What we should have done
related talks
40mWikimedia Content API: A Cassandra Use-case #bbuzz
16mThe Wikimedia infrastructure
35mThe What and Why of Cassandra Datamodelling at Scale
29mCassandra - an introduction
31mScale Your Application to Infinity
36mStories of Cassandra in Adobe Audience Manager
05mBret Piatt interviewed at Velocity 2010
57mCassandra for Metadata Storage: Successes and Failures