Fast Data on-Ramp with Apache Pulsar on K8

on YouTube ↗14m

" summary

Shows an actual NiFi -> Pulsar pipeline that ingests REST and database sources with no code, pushes them into Pulsar topics, runs lightweight Pulsar functions (Java/Go/Python) for per-event transforms, then hands batches to Spark ETL and continuous Flink SQL. Describes deployment: brokers + BookKeeper storage that scales independently, tiered storage to S3, Kubernetes operators/Helm, managed cloud with free tier, support for Kafka/MQTT protocols, millions of topics and real-world deployments at hundred‑petabyte scale.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 0:15Apache NiFi Benefits
3. 1:00Apache Pulsar Benefits
4. 1:53Pulsar Cluster
5. 2:43Offloader & Tiered Storage
6. 3:15Function Mesh
7. 4:03Messaging and Streaming Usage
8. 5:03Apache Pulsar Kafka K8
9. 6:04Apache Pulsar Resources
10. 6:47Auto Scaling Pulsar FN with K8
11. 7:43Apache NiFi Pulsar Connector

" related talks

• 41mDeep Dive into Building Streaming Applications with Apache PulsarTimothy J Spann · CloudOpen @ Open Source Summit Europe 2022
• 35mStreaming Data Made EasyTimothy J Spann, David Kjerrumgaard · OSA Con 2022
• 34mEvent Streaming in the Cloud Native World with Apache PulsarMary Grygleski · Cloud Builders: Java Conf 2023
• 48mApache Pulsar: Event Streaming in The Cloud Native WorldMary Grygleski · GOTO Chicago 2023
• 46mFLiP Into Pulsar AppsTimothy J Spann · Big Data Conference Europe 2022
• 39mCodeless Pipelines With Pulsar and FlinkTimothy J Spann · Data Con LA 2021
• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 30mCreating a Data Fabric for IoTKarthik Ramasamy · Scale By The Bay 2018