Building a Realtime Receiver with Spark Streaming
summary
Builds three concrete Spark Streaming receivers: a simple foreign-exchange HTTP API (used to join GoFundMe donation currency data and to backfill via file input), a Twitter connector that handles rate limits (Twitter often limits to 15 calls per 15 minutes and demo uses a 20s poll), and a queue-based pattern (RabbitMQ) with Redis used to persist state. Shows difference between non-blocking store and blocking store, implements a reliable receiver pattern, demoed a live word-cloud, and covers checkpointing and write-ahead logs.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:23Picking a Wide Receiver
- 0:47Spark Streaming Receivers
- 2:22Receiver Examples
- 3:17Batch Spark
- 5:31ReceiverInputDStream
- 5:41TrueFX Forex Web API
- 6:06Simple HTTP Connector
- 8:04Simplest Receiver
- 8:21What exciting things can we do with just a trickle?
- 10:59Simple Stream
- 11:26Limitations with Simple Web API Streaming
- 13:42Unreliable vs Reliable vs Client-State Reliable Receiver?
- 14:07Twitter API
- 15:06TwitterConnector
- 16:49Let's go look at the demo!
- 18:14RabbitMQ Reliable Receiver
- 18:46Honorary Mention
related talks
43mConnecting Reactive Applications with Fast Data Using Reactive Streams
35mStreaming and Real-Time Analytics
42mPrinciples and Patterns for Streaming Data Analysis
48mTaming & Composing High Performance Stream-Oriented Processing Systems
30mFive Challenges in Real-time Stream Processing and Five Solutions
28mSweet Streams Are Made Of These
01h 00mIn Depth Kafka Streams
53mIntroduction to data streaming