Collecting and Processing OTEL Data at Scale
Talkat Observability Day North America 2024 · on Tuesday, 2024-11-12· at 10:15· Salt Palace | Level 2 | 255 E·
summary
Claims that a stack built on Fluent Bit and OpenTelemetry processes petabytes a day for large orgs, including a LinkedIn use case, by adding a dedicated telemetry pipeline layer that performs enrichment, reduction, redaction and routing. Architecture runs processing at edge, pipeline and backend as appropriate, handles hundreds of thousands of hosts, removes noisy debug traffic to cut cost, and operates in production at multi‑terabyte/petabyte scale.
AI-written summary · about
related talks
28mBuilding Observability Pipeline with Fluent-Bit
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
05mTelemetry Pipelines: Your Recipe for Success
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
11mLightning Talk: OTel Me How to Build a Data Pipeline for Observability
21mBuilding Composable OTel Pipelines: CI/CD, Testing, Team-First, and Sca...
11mObservability 2.0 - Much more than just logs,metrics and traces
40mDeploying OpenTelemetry Organizationally: From Proof of Concept to In-Production at Scale