Elasticsearch 5 or and Bust
summary
Upgrade from Elasticsearch 2 to 5 breaks a critical Kenna Security search backend: a 21-node cluster spikes CPU and crashes because a bug turned filters into scoring so non-scoring queries suddenly do expensive work. Rolling back requires standing up a fresh ES2 cluster and copying data — about five days to copy data — so team rebuilds, posts on Elastic discuss, and a core developer supplies a patch and workaround that stabilizes the cluster. This is a concrete failure, fix, timing, and impact.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:05What is Elasticsearch
- 2:23Elasticsearch lingo
- 3:54The upgrade
- 11:18Cluster stabilized
- 12:22Have a rollback plan
- 13:26Do performance testing
- 14:52Dont ignore small warning signs
- 16:16Lessons learned
- 17:26Use the community
- 18:30Leadership and management
- 19:04Engineering VP
- 21:36Team
- 23:22Summary
- 24:59Bonus Lesson
- 25:58Conclusion
- 26:36Question
related talks
41mElasticSearch (R)Evolution — You Know, for Search ... #bbuzz
06mNever ever upgrade
23mDiving through RCAs a quest for reliability
40mHow to SRE When Everything's Already on Fire
54mScale Your Elasticsearch Cluster
38mMigrating from Self-Managed ElasticSearch to OpenSearch
32mUnleash Your Cluster's Potential with OpenSearch Benchmark
41mScaling Observability: From Chaos to Clarity with Open Source Tools