Telemetry Analysis for Scaling
summary
Circonus describes building a scalable telemetry pipeline: collecting, storing and analyzing high-volume metrics. It covers their architectural evolution including migrating from PostgreSQL to a custom columnar store, switching message transport from RabbitMQ to Fq, and operational lessons learned from failure scenarios during distributed collection and storage.
AI-written summary · about
chapters (13)
- 0:00Telemetry analysis intro
- 1:53Collection and aggregation
- 3:41PostgreSQL storage issues
- 7:19Custom distributed database
- 12:48Rack-level failures handling
- 15:41Parallel testing old/new
- 18:54Embedding analytics in DB
- 20:21Real-time data needs
- 22:09RabbitMQ message issues
- 26:25New message queue design
- 31:24Handling duplicate messages
- 33:12Testing and redundancy
- 34:15Q&A
related talks
19mZero to Scale: Telemetry pipeline with Apache Cassandra
52mDistributed IoT Telemetry Storage Based on PostgreSQL and TimescaleDB
41mDesign (Failures) Case Study
49mThinking in time series, the new solution to your pro...
28mThings I wish I knew before we decided to migrate our metrics infrastructure..
22mPractical approach to building real-time analytics for cybersecurity applications
29mFrom Statsd To OpenTelemetry: Atlassian's Metrics Platform Migration…
48mEvolution of the Metrics Pipeline: How the Architecture Scaled with Load