Ingesting Observability Data: OpenTelemetry Collector or Data Prepper?
summary
Compares OpenTelemetry Collector and Data Prepper with concrete tradeoffs: Collector is Go-based, low-memory and ideal as a sidecar/daemonset with low memory, but lacks dead-letter queues and consistent end-to-end acks; Data Prepper is Java, supports Kafka-backed durable buffer and tail sampling and stateful aggregations, plus DLQs, index management and service map aggregation. Recommends a hybrid: Collector batches and forwards via OTLP to Data Prepper, align batch sizes to ingest buffers, and choose Collector-only for simplicity or Collector+Data Prepper for scalable, HA centralized ingestion (SAP runs ~17,000 instances).
AI-written summary · about
chapters (13)
- 0:00Observability data ingestion
- 1:45OpenTelemetry Collector overview
- 3:10How OpenTelemetry Collector works
- 5:20Strengths and weaknesses of OpenTelemetry Collector
- 7:07OpenTelemetry Collector improvements
- 9:13Data Prepper overview
- 11:20Strengths and weaknesses of Data Prepper
- 14:35OpenTelemetry vs Data Prepper
- 15:43Hybrid architecture
- 17:29Integrating Collector with Data Prepper
- 19:34Choosing the right tool
- 20:38OpenSearch observability stack
- 22:02Q&A
related talks
26mUsing Data Prepper for Observability Ingest
23mControlling Data Overhead with the OpenTelemetry Collector
28mControlling data overhead from the OpenTelmetry Collector
24mBuilding Resilient Telemetry Pipelines: Mastering the OpenTelemetry Collector's Per...
25mOpenTelemetry Collector Deployment Patterns
36mShould I use OTel (collectors), or is Prometheus good enough?
29mOpenTelemetry With OpenSearch: Use-Cases, Status and Roadmap
13mManaging Observability Data at the Edge with the OpenTelemetry Collector and OTTL