Latency goes sub-second in Apache Spark Structured Streaming.
summary
Project Lightspeed implements offset management and moves progress tracking, state checkpointing and log purging off the critical path by making them asynchronous, adds execution pipelining and richer connectors and Python state APIs. Benchmarks show end‑to‑end batch duration falling from ~700–900 ms down to 150–250 ms at 100K–1M events/sec (68–75% or up to 3x improvement) and synthetic in‑engine tests drop a 337 ms batch to ~31 ms; stateful pipelines see ~20–30% latency reduction.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 1:54What is stream processing
- 5:04Spark Structured Streaming overview
- 8:38Key benefits of Structured Streaming
- 11:10Rise of sub-second latency needs
- 16:04Project Lightspeed goals
- 18:53Predictable low latency improvements
- 23:50Performance benchmark results
- 25:17Stateful query optimizations
- 26:42Enhanced functionality updates
- 27:04Python support parity
- 30:13Troubleshooting and connectors
- 31:15Closing and Q&A
related talks
44mReal-time Data Processing with Apache Spark Structured Streaming
08mReal-Time Data Systems: Speed, Scale, and Resilience
01h 01mApache Spark Structured Stream: A Near-Real-Time Data Grinder
24mTaming Data Skew in Production ML Pipelines
44mData Streaming: Which horse to bet on?
41mSpeeding Up Data Pipeline Construction in Apache Spark
36mNo more struggles with Apache Spark (PySpark) workloads in production
28mBest of Stream + Batch: When Streaming Analytics Isn’t