Going deep with Spark Streaming #bbuzz
summary
Spark Streaming exposes micro-batches of RDDs where input happens via receivers (must be serializable) that call store and replicate blocks so sources can be ACKed; reliable vs unreliable receivers determine data-loss guarantees. Windowing uses sliding windows only (window length + slide interval); incremental windows need an inverse function to avoid full reaggregation. Stateful ops use updateStateByKey with checkpointing (recommend checkpoint interval 5–10× the slide), and exactly-once with Kafka + WAL is achievable.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:31About Me
- 2:48What is Spark Streaming?
- 3:17High-level Architecture
- 6:07Custom Input DStream
- 6:27Custom Receiver
- 7:07Receiver Reliability
- 8:08Input DStream and Receiver
- 10:06Creating a DStream via Transformation
- 11:01Transforming the input - Standard Operation
- 13:13Using update StateByKey
- 16:00Joining streams
- 17:20Outputting data
- 18:48From Streaming Program to Spark jobs
- 20:31Tumbling (Count) Windowing
- 21:42Tumbling (temporal) Windowing
- 23:22Non-Incremental Sliding Windowing
- 29:19Recovery
- 31:05Fault Tolerance
related talks
25mCONTINOUS APPLICATIONS WITH SPARK 2.0
42mDistributed Real-Time Stream Processing: Why and How
01h 21mStructured Streaming with PySpark and Azure Databricks
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
42mColliding worlds or bending Spark towards enterprise needs
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
30mLive Coding: Building Stateful and Scalable Reactive Streams...
32mBig Data Processing : Where do we go next ?