Introduction to Data Streaming
summary
Argues that streaming replaces slow hourly ETL with near‑real‑time event processing, using Hazelcast Jet to read an OpenData GTFS endpoint, split trip updates, enrich with static trip and stop_time tables, convert schedule hours to timestamps, and write live positions into a Hazelcast IMDG map. Demo (recorded) shows Swiss trains in red and buses in green on a map using simple interpolation, proving the pipeline works; tradeoffs discussed include windowing, stateful enrichment, and memory‑friendly per‑event processing.
AI-written summary · about
chapters (14)
- 0:00Introduction to speaker
- 1:02Why streaming matters
- 6:25Batch processing limits
- 9:18SQL databases shortcomings
- 11:50Event-driven programming
- 13:41Infinite data sets
- 16:15Near real-time benefits
- 18:24Complex event processing
- 20:12Stream processing tools
- 21:59Hazelcast Jet overview
- 24:29Open data and GTFS
- 26:43Demo architecture
- 27:27Live data visualization demo
- 28:29Talk summary and resources
related talks
53mIntroduction to data streaming
42mPrinciples and Patterns for Streaming Data Analysis
50mStreaming Data: Don’t Miss the Train, Don’t Wait in Vain
48mSwiss Transport in Real Time: Tribulations in the Big Data Stack
29mData Mesh Streaming Patterns: A Real-time Perspective
30mFive Challenges in Real-time Stream Processing and Five Solutions
32mBig Data Processing : Where do we go next ?
14mStreaming Data Pipelines to Enhance Real-Time Decision-Making