Pulsar Functions Deep Dive

on YouTube ↗36m

" summary

Pulsar Functions implements lightweight serverless stream processing on Apache Pulsar, running function instances as threads/processes/pods (including Kubernetes) with runtimes selected per language and gRPC/protobuf used for control. Deployments require tenant/namespace/name, input/output topics, parallelism (e.g. 5–20 instances), and processing guarantees; the system uses a metadata topic as single source of truth (BookKeeper-backed) with first-writer-wins updates and an assignment topic for scheduling. Known limits: metadata growth and slow worker start. Roadmap includes function mesh, dynamic runtime selection and metadata compaction.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 1:30Pulsar Functions:- A Deep Dive Agenda
3. 2:13Pulsar Functions:- A Brief Introduction
4. 6:53Pulsar Functions:- Submission Workflow
5. 17:44Pulsar Functions:- Scheduling Workflow
6. 23:16Pulsar Functions:- Execution Workflow
7. 26:54Pulsar Functions:- Java Instance
8. 29:00Pulsar Functions:- Future Work

" related talks

• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 47mEvent Streaming for the Best of All WorldsMary Grygleski · Spring I/O 2022
• 53mEvent Messaging and Streaming with Apache PulsarMary Grygleski · Build Stuff 2022
• 28mBuild your next-generation stream platform based on Apache PulsarNeng Lu · Flink Forward Virtual 2020
• 45mScalable distributed messaging&streaming with Apache PulsarJulien Jakubowski · Berlin Buzzwords 2023
• 48mApache Pulsar: Event Streaming in The Cloud Native WorldMary Grygleski · GOTO Chicago 2023
• 50mApache Pulsar 101: architecture and featuresSteven Le Roux, Alexandre Duval · Devoxx France 2021
• 51mEvent Streaming and Processing with Apache PulsarMary Grygleski · Conf42 Cloud Native 2022