Large Scale Graph Processing & Machine Learning Algorithms
summary
PayPal converts HDFS logs into graphs with MapReduce (current pipeline takes ~two days) then experiments with a scale-up approach using 1.5 billion edges in memory on a single server with Nvidia Pascal GPUs and SSDs plus a graph DB, replacing Apache Giraph MapReduce jobs to cut graph analytics to minutes (PageRank under two minutes). Feature output feeds H2O gradient boosting and an ensemble with deep learning plus active learning, moving ROC from ~0.96 to ~0.98 on 11M transactions with ~500–600 features.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:32What are your models
- 2:35Motivation
- 3:51Giraffe
- 8:05Full Stack Approach
- 11:08Hardware
- 12:54Validate
- 13:37Graph Algorithms
- 14:33Runtime Performance
- 16:34Key Learnings
- 17:32Machine Learning Algorithms
- 17:52Machine Learning Pipeline
- 21:09Active Learning
- 22:12Active Learning Example
- 22:56Fraud Example
- 23:22Active Learning Framework
- 24:03Experiments
- 25:11Conclusion
related talks
43mLarge Scale Online Graph on Aerospike at PayPal Risk
14mBig Data Processing with GPUs
39mThorny path to the Large Scale Graph Processing
56mGraph Analytic Systems
27mData @scale Seattle - Yucheng Low
24mLarge Scale Graph Computation with Apache Giraph
39mThorny path to the Large Scale Graph Processing
37mBig Data Analytics, Machine Learning, Real Time Processing