Introduction to data streaming
summary
Argues that stream processing gives near‑real‑time answers tradeoffs favor slight inaccuracy for speed and shows a working pipeline using Hazelcast Jet with the in‑memory IMDG to ingest GTFS feeds, enrich trip updates with static stop/trip files, and store results in memory. Demo polls a public REST endpoint with a 30‑second rate limit, splits JSON trip updates, joins them to static data, interpolates vehicle positions between stops, pushes map changes over websockets to a JS front end, and visibly runs on a single node.
AI-written summary · about
chapters (29)
- 0:00#JCON2020 www.joon.one
- 7:00Oops
- 8:19Chunking!
- 8:48Big data!
- 11:49Make everything event-based!
- 13:00From finite datasets to infinite
- 13:29Stateful streams
- 14:43Streaming is "smart" ETL
- 15:30Analytics and Decision Making
- 18:11Persistent event-storage systems
- 19:01Kafka
- 19:38Some in-memory stream processing engines
- 22:44Deployment modes
- 24:49Hazelcast Jet
- 26:51Distributed Parallel Processing
- 28:09Some Open Data initiatives
- 28:39Challenges
- 28:52Access
- 29:17Format
- 29:41Standard
- 30:19Data correctness
- 31:00General Transit Feed Specification
- 31:36GTFS static model
- 32:05GTFS dynamic model
- 32:32Use-case: Swiss Public Transport
- 32:51The available...
- 33:48The dynamic data pipeline
- 34:57Architecture overview
- 41:56Recap
related talks
30mIntroduction to Data Streaming
42mPrinciples and Patterns for Streaming Data Analysis
30mFive Challenges in Real-time Stream Processing and Five Solutions
35mStreaming and Real-Time Analytics
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
47mBuilding Modern Data Streaming Apps with Open Source
29mData Mesh Streaming Patterns: A Real-time Perspective
31mBuilding Real-time Pipelines with FLaNK: A Case Study with Transit Data