Design Observability Solution for Multi-Tenancy Environment
summary
Architecture uses a dedicated Prometheus per tenant that remote_writes tenant headers to a central Cortex cluster for a global view and long‑term (months) storage, OpenTelemetry runs per‑tenant collectors to enable trace filtering and transformations (drop noisy URLs/user‑agents) before sending to Jaeger, Fluent Bit Operator generates per‑tenant cluster‑input/cluster‑output configs for log routing, and Kyverno policies auto‑mutate namespaces to provision observability plumbing. The setup replaces custom controllers, scales, and solves the long‑term query pain (months) while keeping tenant isolation.
AI-written summary · about
chapters (14)
- 0:00Observability in multi-tenancy
- 1:09Multi-tenancy in Kubernetes
- 3:57Challenges in multi-tenancy
- 4:20Three pillars of observability
- 5:43Aggregating metrics
- 8:54Global metrics view and storage
- 12:25Tracing in multi-tenancy
- 13:30Aggregating traces between tenants
- 17:33Filtering traces with OpenTelemetry
- 19:02Aggregating logs
- 21:34Kyver as multi-tenancy engine
- 23:40Dynamic observability with Kyver
- 27:18Observability solutions summary
- 29:04Q&A on Prometheus and Cortex
related talks
43mBuilding a Multi-tenant Monitoring System with Grafana, Prometheus and Cortex
40mMulti-tenant Logging with Opentelemetry Collector- 40mMulti-tenant Logging with Opentelemetry Collector
24mTaming Tenancy, Cost and Architecture at Collibra Through OpenTelemetry and Our...
35mUnlocking Kubernetes Observability: Secure, Tenant-Cen...
46mObservability Ecosystem in Kubernetes
28mUnlocking Multi-Cloud Observability: The Case Study of...
24mBanking Observability at Scale