Introduction to Real-Time Big Data with Apache Spark
summary
Shows how to get Spark running from a downloaded distribution to a cluster: spark-submit, ports and memory settings, and launching a local master and worker. Demonstrates creating RDDs and DataFrames, building simple ETL pipelines with map/filter/reduce and streaming micro-batches, then exposing where shuffles and partitions cost time and memory. Covers cluster concepts—executors, cluster manager—and the Catalyst optimizer plus concrete tuning tips for reducing shuffle and improving throughput on thousands of partitions.
AI-written summary · about
chapters (5)
- 0:00Introduction to Apache Spark
- 10:36Spark core concepts
- 20:51Spark pipeline demo
- 33:33Performance optimization
- 43:14Community and ecosystem
related talks
01h 00mGetting Started with Spark
52mSolving Analytical Problems using Apache Spark
45mFrom Big Data to Fast Data: Apache Spark
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
38mReal-time analytics with Apache Cassandra & Apache Spark
11mScaling R with Spark
16mPetabytes of data: How to use R at scale
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka