My Logs Cost More Than My Services
summary
adjoe scales from 100 to over 50,000 RPS in seven years and produces 2 billion Kafka events and 4 billion logs with 1 million active Prometheus series so cost drives decisions. Logging moves from Elasticsearch+Kibana to a hybrid pipeline using Fluent Bit to route recent logs to ES and long term to Loki then replaced by OpenObserve, cutting logging spend about 60% and restoring 60 day retention while enabling joins with the data lake. Prometheus cuts custom metric cost ~40%, Tempo replaces X Ray for ~80% savings, and Pyroscope adds continuous profiling to find code hotspots.
AI-written summary · about
chapters (17)
- 0:00Introduction and background
- 2:29System architecture overview
- 5:05Logging infrastructure decisions
- 8:13Initial cost reduction efforts
- 10:41Switching to Grafana Loki
- 13:56Hybrid logging approach
- 14:59OpenObserve for long-term logs
- 16:01Metrics and alerting systems
- 17:27CloudWatch vs Prometheus
- 20:18Distributed tracing importance
- 22:47Tracing with Grafana Tempo
- 24:32Continuous profiling introduction
- 25:54Pyroscope for profiling
- 26:39Practical observability example
- 28:49Final observability architecture
- 29:33Cost savings and benefits summary
- 30:58Q&A session
related talks
35mDriving down the cost of observability
30mObservability is Too Expensive - Chris Cooney
46mConquering Distributed Cloud Logging Without Breaking the Bank
40mLarge-Scale Logging Made Easy
27mObservability is Too Damn Expensive!
27mObservability with Prometheus and beyond
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud
41mScaling Observability: From Chaos to Clarity with Open Source Tools