Apache Spark's New Declarative Streaming Pipelines in Action
summary
Shows a concrete pipeline that processes 8 billion daily flight events using Spark’s Declarative Pipelines donated to Apache Spark in June 2025, not marketing fluff. Walks through the actual code: an open‑source PySpark data source with OpenSky founders to read live ADS‑B, a few lines to declare streaming tables and materialized views, and a live Databricks demo that runs continuously. Demonstrates AI natural‑language → SQL queries against the streaming tables, deployment details, and what fails and how retries and incremental materialized view updates behave.
AI-written summary · about
chapters (10)
- 0:00Spark declarative pipelines intro
- 6:20Why declarative pipelines matter
- 15:29Traditional pipeline challenges
- 23:55Streaming tables explained
- 31:00Demo setup first table
- 34:55Flight data streaming demo
- 37:02Materialized views for analytics
- 39:09Unity Catalog governance
- 42:24AI analytics on streaming
- 43:05Q&A
related talks
55mIntroduction to Apache Spark using Frameless
43mBuilding a community fountain around your data stream
35mKeynote: Stream analytics made real with Pravega and Apache Flink
08mReal-Time Data Systems: Speed, Scale, and Resilience
29mThe need for speed: Data streaming architectures
43mLeveraging Azure PaaS for Real-time Social Media Analysis
40mBest Practices and Optimization with the Spark Engine
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale