Logs as Data
summary
Argues that logs become actionable when emitted as events as maps instead of strings, implemented in Clojure by converting incoming lines to maps, applying streaming map-reduce recipes, and doing two-tier aggregation (many receivers then a merger). Describes Pulse — a distributed consumer/analyzer/aggregator processing Heroku’s event firehose to power a realtime dashboard updated about once a second, with stats like rolling means over 60 seconds and per-second rollups computed from emitted per-second rollups recipes. The system runs in production and drives Heroku visibility.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:38code
- 1:53HTML markup
- 2:03SQL queries
- 2:14build specs
- 7:51data via simple abstractions with a general library
- 20:59maps+seqs : Clojure events : ?
- 22:34event processing services
related talks
45mUsing time-series data to better understand our products & infrastructure
29mApplication Logging
28mLogfire - Uncomplicated Observability
29mLogging the right way
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
55mThe data dichotomy: Rethinking data and services with streams
56mReal Time Data Pipelines
38mLogging as a Painkiller