Query Pulsar Streams using Apache Flink
summary
Integration wires Apache Pulsar storage and schemas into Apache Flink so Flink SQL reads Pulsar topics as tables: namespaces map to databases, topics map to tables, and Pulsar schema versions map fields into Flink types with metadata columns (key, msgId, publishTime). Demo shows registering Pulsar as a catalog and running a streaming query that immediately surfaces produced rows. Connector currently guarantees at-least-once delivery; work targets transactions for end-to-end exactly-once and register Pulsar as a Flink catalog, schema-first mapping, segment API vs pop API.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:27Highlights Multi-Tenant Data System Isolation, ACL, Policies
- 5:31Pulsar Use Cases
- 8:41Pulsar - Pub/Sub
- 9:13Pulsar - Topic Partitions
- 10:26Pulsar - Infinite stream storage
- 12:00Pulsar - Stream as a unified view on data
- 12:30Pulsar - Two levels of reading API
- 13:46Unified data processing on Pulsar
- 17:27Flink 1.9 Integration
- 25:18I Pulsar Schema (1)
- 26:58Schemainfo
- 27:51I Pulsar Schema (4) - Producer
- 28:42Consumer
- 29:17Multi versions
- 30:18Schema Row Toples without scheme or with primitive schemes
- 31:38Schema Examples
- 32:57Pulsar Source
- 33:39Pulsar Sink
- 34:00Pulsar Catalog
- 35:35I Unified Data Stack
- 36:27Future Work
- 39:59Pulsar at Europe
related talks
25mContinuous Queries on Apache Pulsar using Flink SQL
41mDeep Dive into Building Streaming Applications with Apache Pulsar
14mFast Data on-Ramp with Apache Pulsar on K8
35mStreaming Data Made Easy
39mCodeless Pipelines With Pulsar and Flink
29mHello Hydrate! From Stream to ClickHouse with Apache Pulsar and Friends
48mApache Pulsar: Event Streaming in The Cloud Native World
34mEvent Streaming in the Cloud Native World with Apache Pulsar