Beyond MapReduce

on YouTube ↗49m

" summary

MapReduce on Hadoop delivers cost‑effective scalability but proves batch‑oriented not real‑time: joins and iterative graph algorithms are awkward, and MapReduce forces heavy JVM churn and disk I/O. Practical fixes: higher‑level layers (Hive, Cascading/Scalding) reduce boilerplate; Spark speeds analytics by caching in memory; Storm handles low‑latency event streams; Pregel/BSP style systems and Titan address graph workloads. Middle‑sized datasets (1–10 TB) benefit from lighter‑weight, fine‑grained, collection‑centric tools rather than raw Java MapReduce.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 1:48Trends
3. 12:48Word Count
4. 18:40Challenge with Mapreduce
5. 20:35The Google File System
6. 24:50Disadvantages
7. 28:44Dremel
8. 31:18Scalding
9. 34:09Alternatives To Cascading and Scalding
10. 35:31The Spark Code
11. 38:48Bulk Synchronous Parallel

" related talks

• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 45mFrom Big Data to Fast Data: Apache SparkStefano Baghino · Codemotion Milan 2015
• 30mScalability Challenges in Big Data ScienceMikio Braun · Berlin Buzzwords 2012
• 45mSolving Real & Big (Data) Problems Using HadoopEva Andreasson · GOTO Aarhus 2013
• 31mML practices in the industryGopal Malakar · Cypher 2017
• 11mHow Companies are Using Spark, and Where the Edge in Big Data Will BeMatei Zaharia · Strata Conference 2014
• 37mBig Data Analytics, Machine Learning, Real Time ProcessingKai Waehner · Codemotion Milan 2016
• 01h 25mApache Spark for Big Data ProcessingLudwine Probst, Ilayaperumal Gopinathan · SpringOne2GX 2015