Real Time Data Pipelines
summary
A real-time pipeline ingests ~15,000 events/s (~20–25 MB/s), about 1.3 billion events a day, and moves raw JSON from beacon servers into Hadoop, Cassandra, Vertica and an Elasticsearch-backed real‑time analytics layer. Prototype using Node + statsd tops ~300 msg/s; rewriting the receiver in Java plus RabbitMQ reaches ~2,600 msg/s and runs in production with a dozen receivers. Live features include embeddable D3 visualizations, WebSocket maps, a profiler and cross-platform eventing; moving to Amazon Kinesis is planned.
AI-written summary · about
chapters (10)
- 0:00REAL-TIME DATA PIPELINES
- 11:23CASE STUDY
- 13:40BEACON PAYLOAD
- 18:12THE PROOF OF CONCEPT
- 29:23VERSION 1
- 32:30VERSION 2 (JAVA BOOGALOO)
- 37:17ELASTICSEARCH CLUSTERING
- 44:55EMBEDDABLE VISUALIZATIONS
- 49:10CROSS-PLATFORM EVENTING
- 53:34PIPELINE MAP
related talks
34mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports
07mTransforming Big Data into Real-Time Product Insights
41mFast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
45mUsing time-series data to better understand our products & infrastructure
49mStreaming Visualisation
48mSwiss Transport in Real Time: Tribulations in the Big Data Stack