Interactive Session: AMA | Pulsar and Flink for Unified Data Processing
summary
Connector code is already merged into Flink 1.14 and implements Flink’s new Source API with split enumerator and split reader to map Pulsar topic partitions to Flink splits and fetch records; it supports checkpoints for exactly‑once semantics. Pulsar’s tiered storage, multi‑tenancy and geo‑replication enable infinite retention and cross‑region reads. Deployments at Tencent and Bigo use Pulsar+Flink for real‑time payment reconciliation, fraud detection and recommendations, and the integration is production‑ready with follow‑on table/catalog work incoming.
AI-written summary · about
chapters (10)
- 0:00Introduction and panel overview
- 1:07Unified data processing trends
- 4:58Industry challenges driving unification
- 9:50Twitter’s unified processing needs
- 12:59Apache Pulsar overview
- 17:17Why Pulsar and Flink work together
- 22:35Real-world Pulsar-Flink use cases
- 27:33Integration maturity and roadmap
- 31:30Connector implementation details
- 33:56Closing and next steps
related talks
28mBuild your next-generation stream platform based on Apache Pulsar
47mEvent Streaming for the Best of All Worlds
53mEvent Messaging and Streaming with Apache Pulsar
48mApache Pulsar: Event Streaming in The Cloud Native World
50mApache Pulsar 101: architecture and features
51mApache Pulsar 101: architecture, concepts & benchmarks
45mScalable distributed messaging&streaming with Apache Pulsar
36mPulsar Functions Deep Dive