Log Management: From grep to Full-Text Search and Back
summary
Compares concrete tradeoffs: Elasticsearch uses inverted index tokenization and aggregations to make full‑text searches fast at the cost of CPU, disk and memory for indexing; Loki hashes label sets into a stream id and stores compressed chunks on object stores like S3 so targeted regex scans reduce index overhead and cloud cost but slow if labels are wrong; mentions Lucene grep native‑image CLI as a lightweight option and highlights runtime fields and searchable snapshots as practical compromises.
AI-written summary · about
chapters (12)
- 0:00Introduction and tech pendulum
- 3:19Log management evolution
- 4:23Early log tools like grep
- 7:12Splunk centralized logging
- 10:23ELK stack and full-text search
- 13:55Costs of structured indexing
- 14:38Grafana Loki approach
- 17:48Loki queries and trade-offs
- 19:12Current trends and projects
- 24:45Adapting to changing needs
- 25:27Benchmarking tools fairly
- 26:52Conclusion and discussion
related talks
31mLoki: Like Prometheus But for Logs
37mGrafana Loki: Like Prometheus, But for logs.
36mBeyond grep: Practical Logging and Metrics
38mLog Analytics: Understanding Complex Systems at Scale
05mGrafana Loki Log Aggregation for Incident Investigations
53mGetting started with Grafana Loki, a modern logs database
50mFocus Log Management: Choose Your Tool Wisely!
04mOpenSearch Performance and Monitoring