Lambda Architecture with Apache Spark
summary
Explains a concrete Lambda implementation: Parquet batch view of historical Twitter events plus an in-memory Spark Streaming with checkpointing speed layer that merges counts at query time, demonstrating recovery after crashes and zero-data-loss via checkpoints. Uses Kafka/HDFS for ingest, Parquet for batch storage, and in-demo merging yields live counts (example tag goes from 11 to 16 after real-time merges). Covers exactly-once vs at-least-once output semantics, tooling tradeoffs, and a working demo that successfully replays and merges batch and realtime results.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 4:45What is Lambda architecture
- 14:36Lambda architecture components
- 17:47Apache Spark for Lambda
- 19:09Spark Streaming basics
- 26:20Lambda demo on Spark
- 37:08Real-world case study
- 43:27Lambda architecture critique
- 47:22Conclusion and resources
related talks
53mNoLambda: A new architecture combining streaming, ad hoc, machine...
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
39mDelta Lake: Diving into Data Lakes Without the Downsides
52mLambda Architecture for Real-Time Analytics — Risks and Benefits
49mLambda Architecture for High Data Volume and Fast Business Response
47mThe future of data: Apache Spark 3.1, Delta Lake and Koalas
38mGetting Started with Delta Lake
42mBuilding Robust Streaming Data Pipelines with Apache Spark