Wide Event Analytics
summary
Argues for treating logs as structured events, ingesting them (Kafka, Avro/protobuf) into a columnar event store so top-K, high‑cardinality group-bys run at interactive latencies. Details: convert traces (Zipkin/Jaeger) into events, partition by time, shard, and use sampling (for example keep 10% sample while preserving rare events) to cut storage. Empirical targets: a 10 second query budget -> ~10 GB/disk/query, columnar layout gives ~10x field read reduction and another ~10x compression, enabling billions of events to be queried quickly.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:41debugging large scale systens using events
- 6:01a metric is an aggregation of events
- 7:10Prometheus and the Problem with metrics
- 9:25top latency by country
- 9:47how many users are impacted?
- 21:261.3 billion events
- 22:32dynamic sampling
- 24:52vectorized processing
related talks
16mFrom Sampling To Full Visibility: Scaling Tracing To Trillions of Spans
24mCracking the Cost Code: A Cost Optimized Approach to Distributed...
49mTransforming Analytical Data with Event-Driven Architecture
42mMake your logs work for you
45mUsing time-series data to better understand our products & infrastructure
41mStep By Step Guide to Building an Application Analytics System
19mFirst Principles of Observability Systems
38mPostmortem: Evolution of an Event recording solution