Building and Scaling Robust Zero-code IoT Streaming Data Pipelines
summary
Two zero‑code pipelines ingest NOAA tidal JSON via Apache Kafka and Kafka Connect, monitored with Prometheus and Grafana, and target either Open Distro Elasticsearch + Kibana or PostgreSQL + Apache Superset. The Elasticsearch path needs custom ingest pipelines, geopoint mapping and bulk API tuning to reach acceptable write rates; initial runs hit about 1,800 inserts/sec until bulk writes are used. The PostgreSQL path uses a hacked JDBC sink into a jsonb column with GIN index and achieves about 42,000 inserts/sec. Camel Kafka Connector and dead letter topic handling prevent fail‑state crashes from HTTP error payloads.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 1:33What is Insta Cluster
- 2:16Overview
- 3:16Whats the story
- 5:58Kafka Connect
- 7:48Rest Source Connector Configuration
- 8:27Elastic Search Sync Connector Configuration
- 10:48Visualization
- 12:27Mapping
- 13:51Map Visualization
- 14:35JSON Error
- 15:00HTTP Error
- 15:29Error Handling
- 16:03Apache Camel
- 16:50Kafka Connector
- 17:40Kafka Connector Test
- 18:36Option 2 PostgreSQL Apache Superset
- 19:43Problem 1 Open Source Kafka Connect
- 21:27Problem 2 Apache Superset
- 22:17Problem 2 Virtual Data Set
- 22:39Problem 3 Visualization Type
- 24:14Scaling
- 24:30Measurement
- 25:23Comparison
- 28:38Limitations
- 29:46Conclusions
- 31:21Elasticsearch vs PostgreSQL
- 32:18Elasticsearch vs Apache Superset
- 33:00Outro
related talks
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
47mBuilding Modern Data Streaming Apps with Open Source
31mFun with Sensors
39mCodeless Pipelines With Pulsar and Flink
42mBuilding a Data Pipeline with Clojure and Kafka
56mTaming IoT Data: Making Sense of Sensors with SQL Streaming
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
43mBuilding Scalable and Extendable Data Pipeline