An Elasticsearch Cluster Over 200 TB
summary
Describes the architecture and pitfalls of a very large Elasticsearch cluster used to store and serve logs (over 200 TB) for the Odnoklassniki project. Covers practical engineering details for organizing storage and access to developers, operational challenges and lessons learned while running Elasticsearch at that scale.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:52System requirements
- 7:38Elasticsearch architecture
- 11:59Data ingestion
- 14:12Early issues and fixes
- 18:31Master node problems
- 24:57Datacenter removal
- 29:40Current cluster config
- 32:30Project outcomes
- 33:13Q&A
related talks
54m7 Petabytes of Logs in Elastic. How We Did It
44mSurviving Elasticsearch
35mHow to Efficiently Manage Logs in Large-Scale Kubernetes Clusters
46mCentralized logging and sleeping well at night
40mLarge-Scale Logging Made Easy
30mData @Scale Seattle- Sundaram Narayanan
41mScaling Observability: From Chaos to Clarity with Open Source Tools
30mCentralized Log Management