Apache Spark Structured Stream: A Near-Real-Time Data Grinder
summary
Apache Spark Structured Streaming is presented as a scalable, simple API for near-real-time processing and transformation of large transaction volumes. The talk explains an architecture and implementation details used to process over 100 million daily billing-attempt transactions with Spark Structured Streaming, and highlights practical aspects of using Spark Streams to work with and transform huge datasets.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:10Apache Spark
- 7:22Stream processing
- 10:55Structured Streaming
- 13:41Batch to stream
- 17:35Use cases and benefits
- 23:41Break and next steps
- 24:22Practical stream
- 36:32Production architecture
- 42:53Real solution code
- 56:19Lessons learned
- 59:56Closing and Q&A
related talks
44mReal-time Data Processing with Apache Spark Structured Streaming
42mBuilding Robust Streaming Data Pipelines with Apache Spark
48mThe internals of stateful stream processing in Spark Structured Streaming
45mData Stream Processing Concepts and Implementations
41mStreaming: an Alternative to CRUD & Batch Processing
44mApache Spark on Azure - solution (...)
28mBest of Stream + Batch: When Streaming Analytics Isn’t
40mDeep dive and best practices of Spark streaming