Reliable OpenTelemetry at Scale: No Queue, No Problem
summary
Claims a production OpenTelemetry pipeline that avoids central queues, handling ~15 GB/s peak and trillions of events per day by keeping a fast direct path into ClickHouse and spilling overflow to object storage. Uses the OpenTelemetry collector with the failover connector, blob storage S3 overflow with event notifications and stateless gateway autoscaling. Catch-up runs via single-threaded catch-up collectors reading SQS/notifications to reingest blobs; cost examples show ~2.1 TB/month steady and ~$10k for an 8‑hour regional outage.
AI-written summary · about
chapters (13)
- 0:00Reliable OpenTelemetry ingest at scale
- 3:05Why traditional patterns failed
- 6:35Persistent queue limitations
- 8:22Kafka not viable
- 10:30Blob storage as buffer
- 14:50Failover connector paths
- 16:56Final architecture with S3 events
- 20:32Operational challenges scaling
- 23:23Collector crashes and memory
- 25:51Cost analysis of design
- 27:15Trade-offs and takeaways
- 32:10Queues vs blob storage
- 33:15Closing thoughts and Q&A
related talks
24mBuilding Resilient Telemetry Pipelines: Mastering the OpenTelemetry Collector's Per...
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
21mBuilding Composable OTel Pipelines: CI/CD, Testing, Team-First, and Sca...
24mTo Make an SLO, First You Must Invent the Universe: OpenTelemetry Collection From Scra...
26mDistributed Tracing: All the Warning Signs Were Out There!!!
28mOpenTelemetry Is Not a Silver Bullet
25mMaking Observability Work at the Edge