Stream Data Processing for Fun and Profit

on YouTube ↗01h 01m

" summary

Explains concrete trade-offs and a working demo: uses Storm for real‑time ingestion, Spark Streaming for micro‑batches, and shows Kafka Streams as a deployable processing model; processes Twitter’s feed (~6000 tweets/sec average) and persists raw events before enrichment. Builds a pipeline that stores tweets, runs Stanford NLP sentiment scoring (slower than ingestion so it scales out), writes results to Couchbase/Elasticsearch, and drives a live dashboard. Demo runs end‑to‑end and highlights replayable persistence and latency vs throughput tradeoffs.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 1:25Why Streaming?
3. 4:20Streaming Data
4. 6:04Types of Data Processing
5. 7:49All Apache, all the Time
6. 10:05No Love for Microsoft?
7. 10:55Processing Model
8. 15:52Programming Model
9. 18:37API and Expressiveness

" related talks

• 42mDistributed Real Time Stream Processing: Why and HowPetr Zapletal · Scala Days Berlin 2016
• 28mAn Architecture to Tweet Them AllJesús Sánchez · PyData Madrid 2016
• 32mBuilding Real-time Systems with Open Source TechnologiesGian Merlino, Fangjin Yang · Strange Loop 2014
• 49mLet’s Build a Streaming Data Pipeline!Robin Moffatt · DataXDay 2020
• 57mTaming IoT Data: Making Sense of Sensors with SQL StreamingHans-Peter Grahsl · Voxxed Days Zurich 2019
• 34mChasing the Stream Processing UtopiaKartik Paramasivam · Strange Loop 2018
• 51mStream all the things! - life at the end of the firehoseYann Schwartz, Sofian Djamaa · Scala.IO 2014
• 42mBuilding a Streaming (E-health) Data Pipeline: When and How?Zohreh Jafari · PyData Yerevan 2022