Keynote: Apache Pulsar
summary
Splunk replaces Kinesis/SQS/RabbitMQ/Kafka with Apache Pulsar inside Splunk DSP, running billions of messages and supporting cloud and on‑prem deployments. Evaluation covers multi-layer broker/bookie/worker architecture, durability modes (persistent, replicated, transient) via BookKeeper, multi‑tenant quotas, six client languages (including Node.js and .NET), and subscription models (exclusive, failover, shared, key‑shared). Performance tests vs Kafka report 5–250× lower publish latency and 25–50% fewer VMs for comparable throughput. Production work includes a Kubernetes proxy, operator, connectors and sync/async geo‑replication; features and fixes are upstreamed.
AI-written summary · about
chapters (15)
- 0:00Introduction and journey
- 1:06Splunk and streaming needs
- 3:58Evaluating streaming systems
- 5:46Scalability in Apache Pulsar
- 7:54Durability guarantees
- 11:01Fault tolerance and availability
- 14:12Multi-tenancy and isolation
- 15:14Client languages and messaging models
- 19:32Persistence and schema support
- 21:18Kubernetes deployment
- 23:05Operability and disaster recovery
- 24:50Performance and cost analysis
- 33:01Observability and ecosystem
- 35:34Splunk DSP with Pulsar
- 37:44Closing remarks and future work
related talks
34mEvent Streaming in the Cloud Native World with Apache Pulsar
52mPulsar Beam, HTTP streaming over Apache Pulsar
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
24mThe Cost of Apache Kafka
25mOperating Cloud-Native Kafka
48mCloud Native Event Streaming with Kafka & Open Standards
30mUsing Solr unconventionally to serve 26bn+ documents
35mArchitectures and Trends for Event Stream Processing on Azure with Open Source Software