Codeless Pipelines With Pulsar and Flink

on YouTube ↗39m

" summary

Live demo shows Apache Pulsar ingesting MQTT and Kafka messages from an IoT device into topics, NiFi processors consuming and transforming JSON without code, and apache flink exposing a Pulsar topic as a table so Flink SQL runs continuous queries and inserts alerts (example threshold ~43°C). Connectors push to Postgres, InfluxDB, Aerospike or S3, tiered storage and geo‑replication are used, the stack runs on a laptop via containers, and a stock demo emits ~10,000 records/sec — a working codeless pipelines example with protocol translation.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 12:21Tiered Persistent Storage
3. 15:17Reasons Why To Use Pulsar
4. 18:51Flink Cluster
5. 36:59Reading Stock Data off of a Rest Endpoint

" related talks

• 14mFast Data on-Ramp with Apache Pulsar on K8Timothy J Spann · Kubernetes Batch + HPC Day Europe 2022
• 41mDeep Dive into Building Streaming Applications with Apache PulsarTimothy J Spann · CloudOpen @ Open Source Summit Europe 2022
• 29mHello Hydrate! From Stream to ClickHouse with Apache Pulsar and FriendsTimothy J Spann · OSA Con 2021
• 25mContinuous Queries on Apache Pulsar using Flink SQLNan Lu · Flink Forward Global 2021
• 47mBuilding Modern Data Streaming Apps with Open SourceTimothy J Spann · Open AI + Data Forum @ Open Source Summit North America 2023
• 36mHail Hydrate! From Stream to LakeTimothy J Spann · Conf42 Machine Learning 2021
• 47mQuery Pulsar Streams using Apache FlinkSijie Guo · Flink Forward Europe 2019
• 46mSmart Transit: Real-Time Transit Information with FLaNKTimothy J Spann · ApacheCon@Home 2021