Splunk Data Stream Processor
summary
Splunk’s Data Stream Processor routes ingest through Apache Pulsar into Flink jobs compiled from SPL via an AST type checker and planner that emits a Flink job graph, targeting multiple pipelines at ~40 GB/s per pipeline (≈70 million records/sec) and supporting tens of thousands of isolated job clusters. Engineering focuses: reduce JVM GC pressure, turn off output pressure (buffer timeout = -1) to cut allocations, and make pipelines self‑tuning and relentlessly reliable.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:52Background
- 3:06Agenda
- 3:39What is Data Stream Processor
- 5:09User Interface
- 6:35Stream Job
- 7:25Live Pipeline
- 8:50Data Integration
- 9:54Processing Layer
- 12:21Technical Challenges
- 13:23Scaling for throughput
- 17:17Scale challenges
related talks
38mKeynote: The Evolution of Data Infrastructure at Splunk
45mApache Flink — Real-Time Data Processing
36mSources, Sinks, and Operators: A Performance Deep Dive
39mCodeless Pipelines With Pulsar and Flink
36mDynamically Generated Flink Jobs at Scale
14mFast Data on-Ramp with Apache Pulsar on K8
42mApache Flink deep-dive #bbuzz
25mData: Moving with the flow