Improving Bad Partition Handling in Apache Cassandra
summary
Netflix runs 900+ Cassandra clusters storing 12+ PB and 12M req/s and describes a workflow that finds offending keys with profile load, stops them via an operator-controlled deny list, and performs targeted repairs with node tool forcecompact to ignore GC grace for specific partitions. That combo drops SSTables-per-read from ~8–10 back to 1–2, removes tombstone storms without global compaction, lets regions be failed back in safely, and shortens incident time from hours to minutes.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:24Bad partitions at Netflix
- 4:56Types and causes
- 7:02Project goals
- 7:43Identifying problematic partitions with tools
- 10:28Partition profiling in 4.1
- 11:54Upgrade to 4.1
- 14:22Blocking partitions
- 17:13GC grace seconds risks
- 18:37Force Compact compaction
- 20:42Future automation
- 22:28Doomstones and errors
- 22:53Q&A
related talks
46mCassandra Serving Netflix @ Scale
52mHow Netflix Provisions Optimal Cloud Deployments of Cassandra
23mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
44mTechniques Netflix Uses to Weather Significant Demand Shifts
41mOptimising Apache Spark and SQL for Improved Performance
53mApache Cassandra. Inception - all you need to know
43mPartitions for Everyone!
29mCassandra - an introduction