Processing large data with Apache Spark
summary
Claims concrete speedups and real deployments: Apache Spark emphasizes in‑memory caching and shows 100x in-memory speed over MapReduce in the slides, with a benchmark where a 100+ TB job runs ~23 minutes vs ~72 minutes. Ecosystem details list HDFS Cassandra HBase S3 and languages Scala Java Python R, libraries Spark SQL DataFrames MLlib GraphX Streaming and connectors to Kafka/Flume. Slides cite 8,000 nodes / 1 PB job / 1 TB‑per‑hour streaming. Demo runs a streaming wordcount with DAG UI and shows caching and lazy evaluation; code examples use Spark SQL + DataFrames + MLlib.
AI-written summary · about
chapters (11)
- 0:00Introduction to big data
- 1:34Understanding big data
- 6:52Apache Spark overview
- 16:05Spark RDD architecture
- 21:58Cluster mode and execution
- 27:51Practical spark demo
- 30:56Spark SQL and data frames
- 34:51Spark R library
- 36:47Spark streaming
- 38:18Machine learning with Spark
- 39:52GraphX and closing thoughts
related talks
53mTips and tricks for robust big data applications
32mWhy Use Scala in a Bigdata Context?
38mPySpark in Practice
59mSpark 2.0
43mIntroduction to Spark Datasets
03mDBTB INT Andrew Or r
36mScaling data workloads using the best of both worlds: pandas and Spark
42mScala: The Unpredicted Lingua Franca for Data Science