How Visualizing Logs Converted to Traces Can Uncover Performance Problems
summary
Logs from a JMS server and an LDAP auth server get parsed into traces using Promtail and Loki and a small Java tool that emits a custom trace file. Queries (via logcli) isolate 87 connection IDs and reveal growing offsets up to 30–40 seconds, caused by sequential LDAP bind/authentication and nested-group searches. Uploading the JSON traces into Grafana shows connections queuing behind each other, leads to fixes (parallel auth and removing nested group lookups), and the recreated traces confirm the problem is resolved.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:59Background on the incident journey
- 2:28Swiss army knife (observability toolkit)
- 3:47Using the Explore feature
- 6:27Visualizing logs
- 7:52Setup
- 11:23Logcli
- 17:05Spans
- 17:23Custom trace file created with Java code
- 21:07Summary
related talks
51mNavigating Distributed Traces
01h 02mThe Power of Metrics, Logs & Traces with Open Source
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
47mHow I Solved Production Issues with OpenTelemetry (And How You Can Too)
23mHow We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries - Bryan Boreham
30mBeyond OpenTracing
05mGrafana Loki Log Aggregation for Incident Investigations
42mDistributed Tracing Opening the Door to Visualisation and Correlation