Evolution of Observability Tools at Pinterest
summary
Pinterest moves its metrics backend from Graphite to OpenTSDB to ten sharded clusters and then to an in‑memory store called Goku (stores 24h with Gorilla compression) to handle ~1 billion metrics/min and 10x growth; Statsboard hosts ~2,000 dashboards and 16,000 alerts, metrics flow via a Kafka sidecar agent, processing pipelines evolve from Storm → Spark Streaming → JobStream, logs use AWS ELK, tracing uses OpenTracing/Zipkin and PinTrace plus Spinnaker/Canta for automated canary analysis.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:40Pinterest
- 4:50Change Cycle Change
- 7:28Statsboard
- 8:46Logsearch
- 9:37Pintrace
- 14:01Usage Growth
- 14:43Tool Architecture
- 17:20Data Reduction
- 20:02TScript Scripting Language for Time-series
- 21:43Integrated Alerts
- 23:09Quick Dash
- 24:05RunDash
- 25:34Automated Canary Analysis
- 27:22Automated Root Cause Analysis
- 32:16Automation Roadmap
- 32:43Lessons Learned
- 35:35Observability Team
- 35:46Acknowledgements
- 36:33We're hiring! Come work with us!
related talks
30mHow we increased developer productivity 2x
48mEvolution of the Metrics Pipeline: How the Architecture Scaled with Load
46mPintrace: A Distributed Tracing Pipeline
30mThe Challenges of Managing Open Source Infrastructure at Bloomberg
26mKeynote: 10 Million Spans Per Second: Lessons From Scaling OpenTelemetry at Reddit
41mScaling Observability: From Chaos to Clarity with Open Source Tools
12mEngineering Productivity, Infrastructure
24mWhat Observability Can Learn From BI