Building Observability with Elasticsearch: Practices for High Load

on YouTube ↗40m

" summary

Practical guidance for running Elasticsearch as an observability backend under high load, covering index and shard layout, ILM (index lifecycle management) policies, Logstash persistent queues, metric downsampling, and monitoring the observability system itself. Shares real-world experience operating a platform handling terabytes of logs and millions of events per day.

AI-written summary · about

" chapters (19)

1. 0:00Monitoring problems
2. 2:14Unified observability platform
3. 5:32Logstash queue data loss
4. 8:47Queue settings and space
5. 10:16Heap memory instability
6. 11:21Index design and shards
7. 14:16Dynamic mapping effects
8. 16:48Index naming standards
9. 19:23Kibana dashboard benefits
10. 20:29Hot/cold/frozen tiers
11. 23:00Metric downsampling savings
12. 24:49Data retention monitoring
13. 27:00Observability for platform
14. 28:44Critical Elastic metrics
15. 31:15Platform building advice
16. 33:04Q&A: field mapping
17. 35:34Q&A: CPU and RAM
18. 36:40Q&A: Logstash config
19. 38:52Q&A: field deprecation

" related talks

• 10mEffective Observability: Best Practices with ElasticEvelien Schellekens · Voxxed Days Brussels 2023
• 41mScaling Observability: From Chaos to Clarity with Open Source ToolsKrishnaswamy Subramanian, Shakir Shakiel · XConf India 2024
• 44mSurviving ElasticsearchVitaliy Kharytonskiy · Highload fwdays'18
• 29mHow (not) to Scale Elasticsearch for Data Analytics!Aishwarya Sankaravadivel · Developer Summit 2020
• 01h 01mBest Practices With ElasticsearchRoy Russo · Devnexus 2020
• 24mOperating at scale with Elastic Search in productionSubramaniajeeva Kandasamy · ODSC India 2018
• 52mGrowing to full-suite observability with Elasticsearch & the ELK Stack (PRT267)David Hope, John Haskell · AWS re:Invent 2022
• 10mWhy your observability is costing you too much?A. Mechain · Kubernetes Community Days France 2023