Large Scale Graph Computation with Apache Giraph

on YouTube ↗24m

" summary

Apache Giraph is an Apache project that implements Pregel-style BSP for graphs as a map-only Hadoop job coordinated by ZooKeeper with one master and many workers, it expresses algorithms as per-vertex compute steps and trivially implements PageRank and max-value, claims about a 5x speedup over iterative MapReduce, previously scales to ~150 million objects on ~400 mappers before RPC limits, moves from Hadoop RPC to netty RPC, uses checkpointing for fault recovery, and still requires the whole partition to fit in memory so Apache Giraph and PageRank work well but large-scale fault tolerance and on-disk scaling remain work to do.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 6:00Aggregators Combiner
3. 6:23Combiners User-defined function to combine messages before being sent or delivered Similar to combiners in Hadoo - Saves on network or memory
4. 6:44Checkpointing
5. 8:57Single Map-Only Job
6. 14:53Even more improved RPC
7. 15:15Improved robustness
8. 15:29YARN!
9. 16:36Monitoring and metrics
10. 17:00Higher-level languages

" related talks

• 39mThorny path to the Large Scale Graph ProcessingAleksey Zinovev · HighLoad++ 2014
• 39mThorny path to the Large Scale Graph ProcessingAleksey Zinovev · HighLoad++ 2014
• 26mLarge Scale Graph Processing & Machine Learning AlgorithmsVenkatesh Ramanathan · MLconf Atlanta 2017
• 30mScalable Graph-Parallel Inference Algorithms and SystemsJoseph E. Gonzalez · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)
• 50mBig Data on Smallish MachinesGuy Blelloch · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)
• 40mKeeping identity graphs in sync with Apache SparkRuben Berenguel · Data Love 2021
• 21mMaja Kabiljo, Software Engineer, Facebook Inc.Maja Kabiljo · MLconf ATL 2015
• 53mMapReduce AlgorithmicsSergei Vassilvitskii · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)