Banking Observability at Scale
Talkat Observability Day Europe 2023 · on Tuesday, 2023-04-18· at 15:50· Hall 7, Room E | Ground Floor | Europe Complex·
summary
Productized open-source into a banking-grade Monitoring Data Pipeline (MDPL) that runs thousands of tenants by sharding Prometheus and Grafana on Kubernetes, using 5,000–6,000 Prometheus instances and ~1,700–2,000 Grafana instances, Envoy for load balancing, Mimir (monolith on VMs) as long-term store with S3-compatible storage, Kafka for transport, and a security proxy plus AD/OpenID access control. Control plane uses a resource-manager (touchpoint) with custom CRDs, OPA policies and 4-eyes approvals; demo shows CLI provisioning, Prometheus → Mimir via remote_write, and dashboards wired automatically.
AI-written summary · about
chapters (11)
- 0:00Banking observability intro
- 1:04Legacy to self-service observability
- 2:29Observability architecture design
- 3:32Kubernetes sharding strategy
- 5:15Mimir and S3 availability
- 6:59Grafana security proxy
- 8:44Data flow to storage
- 9:04Prometheus Grafana service
- 13:01Self-service observability demo
- 19:47Mimir and OpenTelemetry plans
- 20:50Q&A
related talks
32m“Gluing it all together” aka building a platform around Prometheus
43mBuilding a Multi-tenant Monitoring System with Grafana, Prometheus and Cortex
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
28mImprove Monitoring and Observability for Kubernetes with OSS Tools
36mObservability at Scale With Monitoring as Code...
35mUnlocking Kubernetes Observability: Secure, Tenant-Cen...
24mPractical Kubernetes Monitoring with Prometheus
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus