Exclusive Producer: Using Apache Pulsar to build distributed applications

on YouTube ↗31m

" summary

Apache Pulsar adds an exclusive producer mode (Pulsar 2.8) that exposes BookKeeper-style fencing to apps: a topic epoch counter in topic metadata increments when ownership changes, causing older producers to get fenced and any pending writes rejected. Two access modes exist (fail-fast or wait-for-exclusive). Using exclusive producers plus writing decisions to the same topic yields linear, non-interleaved histories and lets teams implement leader election, distributed locks and atomic broadcast on Pulsar without ZooKeeper/etcd, with fencing enforced by brokers.

AI-written summary · about

" chapters (11)

1. 0:00Introduction to exclusive producer
2. 2:16Distributed locks patterns
3. 5:27Leader election concepts
4. 9:24Coordination services comparison
5. 12:12Challenges in distributed systems
6. 16:24Fencing for consistency
7. 19:57Exclusive producer feature
8. 24:12Topic epoch implementation
9. 27:22Building leader election
10. 29:26Distributed locks with Pulsar
11. 30:29Summary and Q&A

" related talks

• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 45mScalable distributed messaging&streaming with Apache PulsarJulien Jakubowski · Berlin Buzzwords 2023
• 45mApache Pulsar: Finally an Alternative to Kafka?Julien Jakubowski · Devoxx France 2024
• 50mApache Pulsar 101: architecture and featuresSteven Le Roux, Alexandre Duval · Devoxx France 2021
• 47mEvent Streaming for the Best of All WorldsMary Grygleski · Spring I/O 2022
• 53mEvent Messaging and Streaming with Apache PulsarMary Grygleski · Build Stuff 2022
• 28mBuild your next-generation stream platform based on Apache PulsarNeng Lu · Flink Forward Virtual 2020
• 48mApache Pulsar: Event Streaming in The Cloud Native WorldMary Grygleski · GOTO Chicago 2023