Practical Cloud Pipelines
summary
A home system uses a Raspberry Pi and four capacitive soil sensors (I²C limit = 4) to produce moisture and temperature events every 5 seconds into Apache Kafka on Confluent Cloud. Two Avro schemas store plant metadata and readings; ksqlDB joins the readings stream to the metadata table, windows counts over one-hour non-overlapping buckets and emits an alert when 240 low readings occur (≈20 minutes of low values). Kafka Connect pushes formatted messages to Telegram for phone notifications; the pipeline runs in production.
AI-written summary · about
chapters (18)
- 0:00Intro
- 3:07Project Scratch Pad
- 4:51Paradigm Shift
- 6:25Thinking in Events
- 9:33Kafka Storage
- 12:36Kafka Cluster
- 14:07Confluent
- 17:13The Hardware
- 19:41The Software
- 20:33Know Your Data (and Plants)
- 22:22Writing Data to Kafka
- 24:42Producing Data
- 25:59Processing with Kafka
- 29:42Creating a Table
- 31:48Creating a Stream
- 32:27Stream Enrichment
- 36:47Windowing and Computing Alerts
- 40:18HTTP Sink Connector
related talks
53mPractical Pipelines: Houseplant Soil Alerting System with ksqlDB
35mPractical Pipelines: A Houseplant Alerting System with ksqlDB
52mPractical Pipelines: A Houseplant Alerting System with ksqlDB
01h 05mPractical Pipelines: A Houseplant Soil Alerting System with ksqlDB
31mPractical Pipelines: A Houseplant Alerting System with ksqlDB.
31mFun with Sensors
56mTaming IoT Data: Making Sense of Sensors with SQL Streaming
20mSet your data streams in motion in the ocean!