CONTINOUS APPLICATIONS WITH SPARK 2.0
summary
Spark 2.0 adds continuous applications that read Kafka and file sources into DataFrames/Datasets, run SQL transforms, windowed aggregations and ML pipelines, and produce outputs with exactly-once end-to-end semantics when sources are traceable and checkpointing is enabled. A loan‑management demo trains a RandomForest on windowed data using watermarking window aggregations so training spans multiple microbatches, and classifies loans; it works but Spark 2.0 is alpha: Kafka sinks, some output modes, certain joins and some ML stages are not yet supported.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:26Continuous Applications
- 0:59Agenda
- 1:50Case study: loan management
- 3:16Why Spark?
- 4:02Spark Streaming
- 6:48Spark 2.0
- 7:50Output
- 13:24Stream Transformations
- 14:58Spark ML Pipelines
- 16:33Checkpointing & Delivery Guarantees
- 18:30Time window aggregations
- 19:16Advanced Stream Manipulations
- 19:49Late data & watermarking
- 21:25Joins
- 22:54What's missing?
- 23:36Summary
- 24:35Thank you for your patience
related talks
59mSpark 2.0
34mGoing deep with Spark Streaming #bbuzz
19mUpdates on Spark, MLflow, and the broader ML ecosystem
54mWhat Does the Future Spark Have in Store for Us?
01h 25mApache Spark for Big Data Processing
01h 48mBig Data made easy with a Spark (3.0)
44mData Science at Scale with Apache Spark
39mMachine Learning with Scala on Spark