Hail Hydrate! From Stream to Lake Using Open Source
summary
End-to-end demo that streams IoT, stock REST calls and a NOAA ZIP of XML into a lake using Apache Pulsar, Apache NiFi and Flink; shows MQTT and Kafka protocol bridges, a Pulsar schema registry, SQL-on-streams tables and automatic sinks to JDBC/S3. A laptop-scale demo writes ~78 records to Postgres, splits thousands of weather XML files into per-airport JSON, and runs continuous Flink queries — the pipeline runs and recovers despite local resource limits.
AI-written summary · about
chapters (6)
- 0:00Start
- 1:18Agenda
- 1:49Get Data into a Data Lake
- 10:35Nifi
- 17:45Live Code
- 23:57Get Data from a Rest Endpoint
related talks
42mStreamhouse Architecture with Flink and Paimon
54mApache Pulsar Development 101
40mStreaming Geospatial Vector Data with Apache Projects
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
31mFun with Sensors
31mFrom Warehouse to Lakehouse: Data at Rest
57mTaming IoT Data: Making Sense of Sensors with SQL Streaming
29mData Acquisition: Overcoming Challenges