Has This Happened Before? Queryable Memory for Database Logs
summary
Operational logs from a fleet of 300 databases move on‑premises into Elasticsearch ingestion where deterministic Python compaction and local models produce a Git‑backed markdown wiki; cloud models (Anthropic) run curated web research while Nemotron 35 handles fast local extraction. Agents run digest, research and curation loops, track recurrence and mark incidents solved after 8 weeks with no reappearance. Current limits are small local GPUs, strict data anonymization, token/timeouts and investigator review for any remote queries.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:08The problem with operational logs
- 2:47Errors without context or meaning
- 4:09An asynchronous assistant for DBAs
- 4:50Scale: hundreds of databases and large log volumes
- 6:39Architecture and Elasticsearch ingestion
- 7:34From database logs to agents
- 8:25Research and curation loops
- 9:33The research agent
- 11:15Human administrator collaboration
- 11:27The digest and curation agent
- 12:56Keeping people in control
- 13:19Inspecting agent behavior
- 14:17Current investigation examples
- 14:56Fleet-level signals and criticality
- 15:38Context and local inference constraints
- 16:49A structured incident schema
- 17:43The research loop
- 18:13Curated sources
- 19:17Memory and recurrence detection
- 20:28Current limitations
- 21:08Future improvements
- 21:45Closing
related talks
40mTurn 10,994 Notes Into Memory
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
45mManaging Data & Analytics Projects Effectively
42mMake your logs work for you
39mBuilding a Memex in Ember
48mHyper Speed! When Big Data Blooms
47mMake sense of your (big) data using Elasticsearch
15mSaved by In-Memory NoSQL