Tracking and Triggering Pattern with Spark Stateful Streaming
summary
A long‑running Spark Structured Streaming tracker processes metadata only, grouping incoming events with mapGroupsWithState into accumulator objects and firing HTTP or Kafka triggers (e.g., a 1GB trigger threshold or >2k files) to start short‑lived Spark workers that compact, consolidate or clean data. The system achieves exactly-once processing, scales by adding executors, handles failed triggers by retaining state, processes hundreds of thousands of metadata records per minute while workers touch tens of millions of data rows, runs 2–3 years in production with minimal incidents and costs about $25k/year.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:58Adobe Experience Platform overview
- 3:26Control plane vs data plane
- 5:12Tracking and triggering pattern
- 7:18Why Spark Stateful Streaming
- 9:03Tracker architecture components
- 10:48Serialization and augmentation
- 13:17Accumulator row groups
- 15:01MapGroupWithState function
- 16:49Triggering rules and actions
- 18:54Timeline of data processing
- 22:03Complexities and weak points
- 24:10Use cases for the pattern
- 26:19Cost and resource analysis
- 27:25Demo introduction
- 37:34Q&A session
related talks
40mDeep dive and best practices of Spark streaming
48mThe internals of stateful stream processing in Spark Structured Streaming
45mData Stream Processing Concepts and Implementations
44mReal-time Data Processing with Apache Spark Structured Streaming
01h 01mApache Spark Structured Stream: A Near-Real-Time Data Grinder
31mHelp I need a stream processor - learning how to chose ...
42mBuilding Robust Streaming Data Pipelines with Apache Spark
08mReal-Time Data Systems: Speed, Scale, and Resilience