Structured Streaming with PySpark and Azure Databricks
summary
Structured streaming claims to make real‑time “continuous applications” practical by treating streams as tables and keeping computations consistent; the demo uses PySpark + Azure Databricks reading JSON messages from Azure Event Hubs, derives a schema, casts columns, visualizes ops, and writes checkpointed output as Parquet so data is replayable. A working parking garage emulator generates entry/exit events, memory and file sinks show low‑latency updates, and checkpointing restores state, but row‑wise duration calculations hit a partition/window limitation and fail to run.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:11Motivation for streaming data
- 5:00Real-time streaming building blocks
- 11:42PySpark fundamentals
- 18:35Databricks workspace overview
- 29:07Why Databricks simplifies processing
- 39:15Continuous applications
- 46:03Structured Streaming principles
- 47:52Demo: streaming pipeline setup
- 50:24Connecting to Event Hubs
- 53:20Processing streaming data
- 1:03:24Interactive analytics
- 1:11:22Streaming challenges
- 1:14:16Checkpointing and storage
- 1:17:32Stream joins and advanced cases
- 1:21:09Q&A
related talks
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
32mReal-Time Data Streaming with Azure Stream Analytics
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
01h 47mRadanalytics.io streaming applications workshop
39mStreaming doesn’t have to be hard
34mGoing deep with Spark Streaming #bbuzz
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
49mBuilding Advance Analytics pipelines with Azure Databricks