Once upon a time Kafka on Pulsar
summary
Describes building compatibility from Kafka clients to Apache Pulsar by first creating a Rust proxy that parses and translates the Kafka wire protocol, then moving that logic into Pulsar as the KoP (Kafka-on-Pulsar) protocol handler in the JVM. Covers support for Kafka 2.0–2.8 (with early 3.0 tests), adds schema-registry integration, multi-tenancy, geo-replication and tiered storage, and shows trade-offs: the Rust proxy uses much less memory but makes offset/group coordination and proxying transactions harder (transactions were initially in-memory).
AI-written summary · about
chapters (31)
- 0:00Speakers' Introduction
- 1:23Context
- 3:43Introduction to Pulsar
- 4:54What is Pulsar?
- 6:07Pulsar Features
- 8:16Pulsar Limitations
- 13:57Efforts to Integrate Kafka with Pulsar
- 14:46How to Implement Kafka on Pulsar?
- 15:17How to Put the Protocol in Pulsar?
- 15:55Why Write in REST?
- 17:39Protocol Details
- 18:20The State Machine
- 19:08Step Machines
- 19:20Performance
- 22:02Differences Between Kafka and Pulsar
- 23:08Distributed Transaction
- 26:24Collaboration with StreamNative
- 26:59Pulsar Architecture
- 27:33Starting Another Protocol
- 27:46Pulsar Interface
- 28:12Kafka Management
- 30:26Using Kafka Producer
- 30:58Messaging Stack Migration
- 32:05Pulsar Features
- 33:42Auto-Creation of Topics
- 35:16Froid Enhancement
- 35:56Compatibility with Kafka 3.0
- 36:422020 Recap
- 38:06Other Protocols
- 39:17Test Pulsar
- 39:44Q&A
related talks
48mBest of Both Worlds: Apache Pulsar and Apache Kafka
45mApache Pulsar: Finally an Alternative to Kafka?
31mThe Power of Apache Pulsar. Harnessing Dapr to Build High Scale Messaging
47mEvent Streaming for the Best of All Worlds
51mApache Pulsar 101: architecture, concepts & benchmarks
46mScalable messaging and streaming with Apache Pulsar
45mScalable distributed messaging&streaming with Apache Pulsar
28mBuild your next-generation stream platform based on Apache Pulsar