Using time-series data to better understand our products & infrastructure
summary
Claim: product and infra behavior become visible only when events are captured with rich context at high frequency, normalized, and recorded to an append-only log for many consumers. Built a pipeline that instruments ActiveRecord, uses Kafka as a log (not a queue), pushes batched blocks to S3 and query with Hive/Redshift, and runs a local Unix-socket daemon to avoid adding web latency. libpcap captures network flows. Surprises: pre-collection message loss is effectively 100%, aggregation hides important structure (GC cycles and short spikes), and the pipeline does work operationally.
AI-written summary · about
chapters (6)
- 0:00Start
- 21:02a triumph of monitoring and system visibility
- 24:12Event Collection
- 31:37Predictable Event Structure
- 37:02Event Recording "Helpers"
- 39:44Analysis
related talks
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
45mTracing: A journey to tactical insights
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
42mMake your logs work for you
29mData Acquisition: Overcoming Challenges
29mWide Event Analytics
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
07mTransforming Big Data into Real-Time Product Insights