Building a Streaming (E-health) Data Pipeline: When and How?
summary
Argues that streaming is the right choice when data is massive, unbounded, or needs timely answers, and shows an e‑health pipeline that actually runs: a five‑tier design (collection → message queue → processing → in‑memory store → data access), Kafka as the message bus, Spark Structured Streaming for processing, micro‑batches for latency tradeoffs, and in‑memory stores for recent state. Uses it for 10‑minute dashboards, monthly aggregates and real‑time fraud/error detection and reports that the stack meets those SLAs.
AI-written summary · about
chapters (15)
- 0:00Data pipelines overview
- 2:01Real-world pipeline needs
- 5:12DevOps pipeline stack
- 8:06Batch to streaming shift
- 11:41Streaming vs batch choice
- 13:52Streaming vs batch examples
- 16:24E-prescription case study
- 18:30Streaming pipeline blueprint
- 22:03Real-time data definition
- 25:12Message queuing role
- 31:45Kafka delivery semantics
- 33:36Batch vs streaming queries
- 35:07Spark streaming tool
- 38:24Spark Streaming types
- 40:15Post-processing access
related talks
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
47mOne streaming to rule them all: Streaming as the foundation of an analytics ecosystem
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
19mDemystifying Modern Data Pipeline Architecture
49mLet’s Build a Streaming Data Pipeline!
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
31mZero to data infrastructure
50mStreaming Data: Don’t Miss the Train, Don’t Wait in Vain