Creating a Data Fabric for IoT

on YouTube ↗30m

" summary

Apache Pulsar is presented as a single system that combines messaging, storage and stream processing with real deployments: runs on Raspberry Pi and single-node edges, supports millions of topics and claims 2.3 million-device scale at Yahoo, uses a decoupled broker/bookie design for fast recovery, offers Pulsar Functions for lightweight on-edge compute (100–200 MB JVM or ~60–70 MB lightweight footprint on device), and moves old data to tiered S3 storage with cross‑data‑center replication and SQL queries over retained streams.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to IoT data fabric
2. 2:41IoT environment and device tiers
3. 7:01Requirements for IoT data fabric
4. 12:19Apache Pulsar overview
5. 18:42Pulsar features for IoT
6. 23:37Processing data with Pulsar
7. 25:01Q&A and closing

" related talks

• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 55mEvent Messaging and Streaming with Apache PulsarMary Grygleski · Great International Developer Summit 2023
• 59mEvent Streaming and Processing Using Apache PulsarMary Grygleski · IntelliJ IDEA Conf 2022
• 35mStreaming Data Made EasyTimothy J Spann, David Kjerrumgaard · OSA Con 2022
• 41mDeep Dive into Building Streaming Applications with Apache PulsarTimothy J Spann · CloudOpen @ Open Source Summit Europe 2022
• 34mEvent Streaming in the Cloud Native World with Apache PulsarMary Grygleski · Cloud Builders: Java Conf 2023
• 48mApache Pulsar: Event Streaming in The Cloud Native WorldMary Grygleski · GOTO Chicago 2023
• 14mFast Data on-Ramp with Apache Pulsar on K8Timothy J Spann · Kubernetes Batch + HPC Day Europe 2022