Live build: How to harness streaming data in real time
summary
Live demo wires a Bluetooth heart‑rate strap -> phone -> websocket gateway -> Kafka, then runs layered microservices that interpolate timestamps, compute per‑interval calories with a published formula, aggregate session totals in a stateful in‑memory view and expose results on a Dash dashboard. The build uses the Quix platform and Kafka + Kubernetes + Python, stores streams to a side database for history, checkpoints state to disk, uses protobuf for fast messages, claims ~10 ms inter‑service latency, and the pipeline runs end‑to‑end.
AI-written summary · about
chapters (9)
- 0:00Introduction to streaming
- 2:12Batch vs. streaming data
- 3:53System architecture
- 5:58Core stream concepts
- 8:58IoT data pipeline demo
- 15:55Batch processing analysis
- 23:21Streaming transformation demo
- 31:13Stateful processing and UI
- 37:15Real-time dashboard and Q&A
related talks
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
40mBuilding Real-Time Applications: Cyclist Crash Detection
43mSimplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
01h 47mRadanalytics.io streaming applications workshop
28mReal-Time Revolution: Kickstarting Your Journey in Streaming Data
01h 02mBig Data Analytics in Near-Real-Time with Apache Kafka Streams
44mShaping the Future of Real-time Data Pipeline