Monitoring Distributed Java Applications at Scale
summary
A small ticket-order system of 5 microservices is instrumented with Prometheus, Grafana, Jaeger and Loki; traces use custom spans and tags, metrics include a custom event counter and tracing metrics, and logs are routed to Loki with a redaction interceptor. Runtime debugging finds a null pointer caused by an incorrect landmark URL in Kubernetes manifests and a deliberate failure gated by a test flag; both are fixed by editing the deployment and setting the flag. Grafana alarms a service error ratio threshold error ratio >0.05 and dashboards show span counts, durations and refund ratios.
AI-written summary · about
chapters (9)
- 0:00Start
- 5:41pulling metrics from prometheus
- 6:00pull some tracing metrics in prometheus
- 7:55use the tags feature of jager
- 15:18edit directly in the dashboard
- 21:33set alerts on template variables
- 22:25create an alert
- 25:04custom log interceptors
- 29:25observe your critical business operations via custom griffon alerts
related talks
28mImprove Monitoring and Observability for Kubernetes with OSS Tools
28mMonitoring Microservices w. Prometheus & Tracing Apps w. OpenTracing
42mDistributed Monitoring: How to understand the Chaos
52mIs your JVM app flying blind? Unmask hidden issues with Observability superpowers!
35mMonitoring Microservices
24mPractical Kubernetes Monitoring with Prometheus
39mThe Hidden Gems of Distributed Tracing
34mMonitor My Socks: Using Prometheus in a Polyglot Open Source Microservices