Using Data Prepper for Observability Ingest
summary
Data Prepper is an Apache‑2 open source last‑mile collector that takes traces, logs and metrics, integrates with OpenTelemetry collector and Fluent Bit, and performs stateful processing and enrichment before indexing to OpenSearch. A demo shows combining 60 disparate log lines into 10 aggregated documents using the aggregate processor. It also reads S3 objects, supports sampling, redaction and routing, and 2.0 adds multi‑node aggregation, conditional routing, and JSON/CSV parsers.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:30OpenSearch Observability
- 6:05What is OpenSearch Data Prepper?
- 7:28Pipelines
- 9:48Trace Analytics
- 14:38Metrics
- 20:56Customer Stories
- 24:18Getting Started
related talks
24mIngesting Observability Data: OpenTelemetry Collector or Data Prepper?
25mPersistent and performant ingestion using Data Prepper
29mOpenTelemetry With OpenSearch: Use-Cases, Status and Roadmap
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
37mHow SAP analyzes OpenTelemetry signals using Data Prepper
25mUnifying Observability: Weaving Prometheus, Jaeger, and Open Source Together to Win
17mFrom Reactive to Proactive: Event-Driven Architectures for Modern...
24mOne Pane to Rule Them All: Uniting the Prometheus Community with O...