Observability that scales: Analyze data in stream and query remotely
summary
Coralogix processes logs in-stream with Kafka Streams, extracts state into RocksDB and uses hyperloglog for low memory cardinality, then archives dynamic per-key CX data to S3 for remote query; the vendor claims 99% of data never searched, that the system can scan 10 TB in 30 seconds, and that customers can reduce TCO by ~40%. Nutanix ships Kubernetes logs via Fluentd, metrics via Prometheus remote write and Lambdas use a Coralogix extension, keeping hot storage for ~20–30% of data and running alerts and dashboards in-stream.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:32Observability is accelerating
- 3:11Our practices are evolving
- 5:02What is the primary hurdle to overcome?
- 5:47What are we missing in all of this data?
- 11:00A different approach
- 20:06Nutanix insights: Predictive health and support automation
- 21:07Nutanix Insights: Creating customer delight
- 23:03Nutanix Insights server architecture
- 25:53Nutanix Insights infra monitoring requirements
- 31:41Forging a new path to observability
related talks
30mObservability is Too Expensive - Chris Cooney
38mLog Analytics: Understanding Complex Systems at Scale
35mDriving down the cost of observability
13mNext-Gen Observability: reduce MTTR
41mResilient Observability at the Retail Edge: A Lightweight, Scalable, and...
08mMastering Observability: The Coralogix Maturity Blueprint
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
40mLarge-Scale Logging Made Easy